Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Jim Manico]

I love you Neil.

--
Jim Manico
@Manicode

> On Oct 30, 2019, at 3:18 PM, Neil Madden  wrote:
> 
> 
> If you can point out where I recommended disabling TLS or not bothering to 
> strip headers from incoming requests, or anything else along those lines then 
> please let me know. Otherwise, yes we’re done here. 
> 
>>> On 30 Oct 2019, at 17:19, Salz, Rich  wrote:
>>> 
>> 
>> To quote your previous claim: "There is no such thing as an unguessable 
>> name."
>> Right.  That doesn’t mean *I* have to guess it.
>> 
>> Even if your deployment team had such staggeringly bad operational security 
>> practices as to allow people to take packet captures from an internal 
>> network and show them on public slides without any kind of questions being 
>> asked, if this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION 
>> WHERE YOU USED A WELL-KNOWN HEADER NAME*!
>> Yes you are worse off.  Because that now-exposed header value can be used 
>> for spoofing.  As opposed to protection by TLS, and then sending the 
>> plaintext message around.
>>  
>> I don't know how many different ways I can say that this is a defense in 
>> depth
>> Because it is not.  It is taking an application-level piece of configuration 
>> data and requiring it to be treated as if it were crypto material. Which it 
>> cannot be, because multiple parties need to know it (as I said, the proxy, 
>> the backend, the app developers, the support team, etc).  It’s defense by 
>> “collapsing layers” rather than “in depth.”
>>  
>> As with all defense in depth, the aim is to be more than 1 configuration 
>> mistake away from total compromise.
>> But that is exactly what you are proposing. Exposing the header *is* a total 
>> compromise and multiple entities will need to know that header value.
>>  
>> At any rate,  I think we’re done here.
>>  
> ___
> OAuth mailing list
> OAuth@ietf.org
> https://www.ietf.org/mailman/listinfo/oauth
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Salz, Rich]

If you strip off the TLS, you *still* cannot pass messages around without 
protecting them as if they had secrets because, well, the header name you are 
using is secret.  This is not a defense in depth.
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

If you can point out where I recommended disabling TLS or not bothering to 
strip headers from incoming requests, or anything else along those lines then 
please let me know. Otherwise, yes we’re done here. 

> On 30 Oct 2019, at 17:19, Salz, Rich  wrote:
> 
> 
> To quote your previous claim: "There is no such thing as an unguessable name."
> Right.  That doesn’t mean *I* have to guess it.
> 
> Even if your deployment team had such staggeringly bad operational security 
> practices as to allow people to take packet captures from an internal network 
> and show them on public slides without any kind of questions being asked, if 
> this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION WHERE YOU 
> USED A WELL-KNOWN HEADER NAME*!
> Yes you are worse off.  Because that now-exposed header value can be used for 
> spoofing.  As opposed to protection by TLS, and then sending the plaintext 
> message around.
>  
> I don't know how many different ways I can say that this is a defense in depth
> Because it is not.  It is taking an application-level piece of configuration 
> data and requiring it to be treated as if it were crypto material. Which it 
> cannot be, because multiple parties need to know it (as I said, the proxy, 
> the backend, the app developers, the support team, etc).  It’s defense by 
> “collapsing layers” rather than “in depth.”
>  
> As with all defense in depth, the aim is to be more than 1 configuration 
> mistake away from total compromise.
> But that is exactly what you are proposing. Exposing the header *is* a total 
> compromise and multiple entities will need to know that header value.
>  
> At any rate,  I think we’re done here.
>  
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Salz, Rich]

  *   To quote your previous claim: "There is no such thing as an unguessable 
name."
Right.  That doesn’t mean *I* have to guess it.


  *   Even if your deployment team had such staggeringly bad operational 
security practices as to allow people to take packet captures from an internal 
network and show them on public slides without any kind of questions being 
asked, if this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION 
WHERE YOU USED A WELL-KNOWN HEADER NAME*!
Yes you are worse off.  Because that now-exposed header value can be used for 
spoofing.  As opposed to protection by TLS, and then sending the plaintext 
message around.


  *   I don't know how many different ways I can say that this is a defense in 
depth
Because it is not.  It is taking an application-level piece of configuration 
data and requiring it to be treated as if it were crypto material. Which it 
cannot be, because multiple parties need to know it (as I said, the proxy, the 
backend, the app developers, the support team, etc).  It’s defense by 
“collapsing layers” rather than “in depth.”


  *   As with all defense in depth, the aim is to be more than 1 configuration 
mistake away from total compromise.
But that is exactly what you are proposing. Exposing the header *is* a total 
compromise and multiple entities will need to know that header value.

At any rate,  I think we’re done here.

___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

On 30 Oct 2019, at 16:41, Salz, Rich  wrote:
> 
> I'm thinking of a uniformly random 16 byte name right now. Have at it.
> Cute but missing the point.  I don’t have to guess it. 

To quote your previous claim: "There is no such thing as an unguessable name."

> YOU have to securely deploy it across your proxy (however many staff), your 
> backend (however many staff), your application developers (however many), and 
> perhaps your diagnosing or debug teams if they are different.  And then you 
> must make sure that if ANYONE ever takes a packet trace, or makes a slide out 
> of a sample message, that they don’t disclose the header, such as by showing 
> “here’s how we do OAUTH” at a user group meeting.

Even if your deployment team had such staggeringly bad operational security 
practices as to allow people to take packet captures from an internal network 
and show them on public slides without any kind of questions being asked, if 
this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION WHERE YOU 
USED A WELL-KNOWN HEADER NAME*!

I don't know how many different ways I can say that this is a defense in depth 
*in addition to* everything else you would normally do to secure traffic 
between your RP and backend servers. As with all defense in depth, the aim is 
to be more than 1 configuration mistake away from total compromise.


-- Neil___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Salz, Rich]

  *   I'm thinking of a uniformly random 16 byte name right now. Have at it.
Cute but missing the point.  I don’t have to guess it.  YOU have to securely 
deploy it across your proxy (however many staff), your backend (however many 
staff), your application developers (however many), and perhaps your diagnosing 
or debug teams if they are different.  And then you must make sure that if 
ANYONE ever takes a packet trace, or makes a slide out of a sample message, 
that they don’t disclose the header, such as by showing “here’s how we do 
OAUTH” at a user group meeting.

  *   Again, this is a defense in depth measure. A config file is fine.
No, it’s not.  It’s a multi-party shared secret that isn’t identified as a 
secret.  It’s not defense in depth, it’s a foundation of sand.

  *   irrelevant to the current discussion, which is about how the backend 
distinguishes security-critical headers that the proxy set from 
security-critical headers that were sneaked past the proxy by a client (through 
misconfiguration or parsing bug).
The backend must trust the proxy.  If you tell the proxy to “use FOOBAR123” as 
the header of the client certificate, how do you know that the proxy properly 
provided the client certificate?  Pretending that the backend only has to trust 
*part* of the proxy, because we use a sekrit header value is just that, 
pretending.

Your links to confused deputy and CSRF do not address the issues I raised.


  *   Authenticating the other side of a communication pipe is not sufficient 
to authenticate the origin of the data contained within those messages. The 
whole point of a proxy is that it forwards requests from clients. In the face 
of misconfigurations and parsing bugs the backend cannot distinguish headers 
that were set by the proxy from headers that were spoofed by the client. *This 
is the entire problem I have been discussing*.
And you believe that configuring a proxy, of which you are skeptical, addresses 
that concern?

___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

Combining responses to related messages (429 error):

On 30 Oct 2019, at 14:07, Salz, Rich  wrote:
> 
>> But an unguessable header name is *simple* and effective and works right now 
>> with widely implemented functionality. 
> 
> You mean like admin/admin for administrator access?  There is no such thing 
> as an unguessable name.

I'm thinking of a uniformly random 16 byte name right now. Have at it.

> You claim the name will never be exposed to untrusted parties.  How so?  You 
> are now telling administrators to treat a *name* as securely as they treat a 
> *key* (or password).  If it must be protected like key material, then use it 
> like key material.

Again, this is a defense in depth measure. A config file is fine.

> The proxy-backend should be TLS, ideally authenticating the proxy.

I agree - but completely irrelevant to the current discussion, which is about 
how the backend distinguishes security-critical headers that the proxy set from 
security-critical headers that were sneaked past the proxy by a client (through 
misconfiguration or parsing bug).


> On 30 Oct 2019, at 14:18, Salz, Rich  wrote:
> 
> Again, authenticating the *connection* from the RP to the backend services is 
> good, but is completely orthogonal to authenticating the headers themselves.
>  
> I strongly disagree.  Authenticating the sender allows the receiver to make a 
> trust decision in the provenance and quality of the data it gets from the 
> sender.  Do you disagree with that?

Yes, see:

 https://en.wikipedia.org/wiki/Confused_deputy_problem 

https://en.wikipedia.org/wiki/Cross-site_request_forgery 

https://en.wikipedia.org/wiki/Server-side_request_forgery 


and so on and so on. Authenticating the other side of a communication pipe is 
not sufficient to authenticate the origin of the data contained within those 
messages. The whole point of a proxy is that it forwards requests from clients. 
In the face of misconfigurations and parsing bugs the backend cannot 
distinguish headers that were set by the proxy from headers that were spoofed 
by the client. *This is the entire problem I have been discussing*. 

-- Neil___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Salz, Rich]

  *   Again, authenticating the *connection* from the RP to the backend 
services is good, but is completely orthogonal to authenticating the headers 
themselves.

I strongly disagree.  Authenticating the sender allows the receiver to make a 
trust decision in the provenance and quality of the data it gets from the 
sender.  Do you disagree with that?
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

On 30 Oct 2019, at 14:05, Torsten Lodderstedt  wrote:
>> On 30. Oct 2019, at 14:56, Neil Madden  wrote:
>> 
>> On 30 Oct 2019, at 13:24, Justin Richer  wrote:
>>> 
>>> All of these problems can be solved, and I think solved better, by securing 
>>> the connection between the proxy and the back-end. That way the back end 
>>> will be able look not only for a specific header, but verify that the 
>>> header came from the proxy itself. An obscure header name is one way to do 
>>> that, but it has a lot of issues with it, especially since it’s likely to 
>>> be selected once during set-up and never changed throughout the lifetime of 
>>> the deployment. I think there are likely much better solutions here, and 
>>> they’d address this issue without things getting weird.
>> 
>> The issue has nothing to do with the security of the connection between the 
>> proxy and the backend. It is to do with data origin authentication of the 
>> headers themselves. All of the headers arrive at the backend over the same 
>> connection, but only some of them were created by the proxy. There are 
>> undoubtedly better alternatives - e.g. using a shared secret to compute a 
>> HMAC over security sensitive headers and include that as an additional 
>> header or field. But an unguessable header name is *simple* and effective 
>> and works right now with widely implemented functionality. 
>> 
>> There's no need for the header name to ever change - the secret is not 
>> exposed to untrusted parties
> 
> If the proxy sends certs via an header X to service A and B and someone 
> impersonates A it will find out the secret and use it to inject certs in a 
> connection towards B.

Being able to impersonate A suggests that the attacker is already on the 
trusted side of the network and that you are employing zero additional network 
security controls between the RP and service A and B (e.g., TLS, firewalls, 
VLAN/switches, etc). Remember, this is intended to mitigate against accidental 
misconfiguration of the RP and header spoofing tricks coming from the external 
network. I'm not suggesting it as an alternative to basic network security on 
the trusted network.

> We have learned that it is sometime hard to use different secret header names 
> for the same purpose with different request targets. In those cases, a way to 
> authenticate the proxy might by a good solution.  

Do you have an example? The RPs and load balancers I'm familiar that support 
multiple backends all support sending different headers to each of them. Again 
though, security against attackers inside the trusted network is not part of 
what the solution is preventing.

Again, authenticating the *connection* from the RP to the backend services is 
good, but is completely orthogonal to authenticating the headers themselves.

-- Neil


___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Salz, Rich]

> But an unguessable header name is *simple* and effective and works right now 
> with widely implemented functionality. 

You mean like admin/admin for administrator access?  There is no such thing as 
an unguessable name. You claim the name will never be exposed to untrusted 
parties.  How so?  You are now telling administrators to treat a *name* as 
securely as they treat a *key* (or password).  If it must be protected like key 
material, then use it like key material.

The proxy-backend should be TLS, ideally authenticating the proxy.
 

___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Torsten Lodderstedt]

> On 30. Oct 2019, at 14:56, Neil Madden  wrote:
> 
> On 30 Oct 2019, at 13:24, Justin Richer  wrote:
>> 
>> All of these problems can be solved, and I think solved better, by securing 
>> the connection between the proxy and the back-end. That way the back end 
>> will be able look not only for a specific header, but verify that the header 
>> came from the proxy itself. An obscure header name is one way to do that, 
>> but it has a lot of issues with it, especially since it’s likely to be 
>> selected once during set-up and never changed throughout the lifetime of the 
>> deployment. I think there are likely much better solutions here, and they’d 
>> address this issue without things getting weird.
> 
> The issue has nothing to do with the security of the connection between the 
> proxy and the backend. It is to do with data origin authentication of the 
> headers themselves. All of the headers arrive at the backend over the same 
> connection, but only some of them were created by the proxy. There are 
> undoubtedly better alternatives - e.g. using a shared secret to compute a 
> HMAC over security sensitive headers and include that as an additional header 
> or field. But an unguessable header name is *simple* and effective and works 
> right now with widely implemented functionality. 
> 
> There's no need for the header name to ever change - the secret is not 
> exposed to untrusted parties

If the proxy sends certs via an header X to service A and B and someone 
impersonates A it will find out the secret and use it to inject certs in a 
connection towards B. We have learned that it is sometime hard to use different 
secret header names for the same purpose with different request targets. In 
those cases, a way to authenticate the proxy might by a good solution.  

> and is a defense-in-depth rather than a primary defense. I also don't know 
> why you consider this a "weird" solution - it's a simple pragmatic solution 
> to a fairly widespread class of security vulnerabilities. It also has the 
> benefit that it forces the backend to "validate" the secret, because it won't 
> find the header if it gets it wrong, whereas it is much easier to forget to 
> validate a HMAC tag. I'll take simple and weird over missing and broken any 
> day.
> 
>> 
>> And one of the best things about a standard is that you’re still free to 
>> completely ignore it if you want to, so people can and will keep following 
>> whatever proprietary patterns they want to. But at least a standard 
>> mechanism would give us a way to say to newcomers and veterans alike “No 
>> really, here’s a way that we all agree works and has these properties”. 
> 
> I'm not arguing against a standard, just against a standard that makes it 
> harder to mitigate known security vulnerabilities.
> 
> -- Neil
> ___
> OAuth mailing list
> OAuth@ietf.org
> https://www.ietf.org/mailman/listinfo/oauth



smime.p7s
Description: S/MIME cryptographic signature
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

On 30 Oct 2019, at 13:24, Justin Richer  wrote:
> 
> All of these problems can be solved, and I think solved better, by securing 
> the connection between the proxy and the back-end. That way the back end will 
> be able look not only for a specific header, but verify that the header came 
> from the proxy itself. An obscure header name is one way to do that, but it 
> has a lot of issues with it, especially since it’s likely to be selected once 
> during set-up and never changed throughout the lifetime of the deployment. I 
> think there are likely much better solutions here, and they’d address this 
> issue without things getting weird.

The issue has nothing to do with the security of the connection between the 
proxy and the backend. It is to do with data origin authentication of the 
headers themselves. All of the headers arrive at the backend over the same 
connection, but only some of them were created by the proxy. There are 
undoubtedly better alternatives - e.g. using a shared secret to compute a HMAC 
over security sensitive headers and include that as an additional header or 
field. But an unguessable header name is *simple* and effective and works right 
now with widely implemented functionality. 

There's no need for the header name to ever change - the secret is not exposed 
to untrusted parties and is a defense-in-depth rather than a primary defense. I 
also don't know why you consider this a "weird" solution - it's a simple 
pragmatic solution to a fairly widespread class of security vulnerabilities. It 
also has the benefit that it forces the backend to "validate" the secret, 
because it won't find the header if it gets it wrong, whereas it is much easier 
to forget to validate a HMAC tag. I'll take simple and weird over missing and 
broken any day.

> 
> And one of the best things about a standard is that you’re still free to 
> completely ignore it if you want to, so people can and will keep following 
> whatever proprietary patterns they want to. But at least a standard mechanism 
> would give us a way to say to newcomers and veterans alike “No really, here’s 
> a way that we all agree works and has these properties”. 

I'm not arguing against a standard, just against a standard that makes it 
harder to mitigate known security vulnerabilities.

-- Neil
___
OAuth mailing list
OAuth@ietf.org
https://www.ietf.org/mailman/listinfo/oauth

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Torsten Lodderstedt]

+1

> On 30. Oct 2019, at 14:24, Justin Richer  wrote:
> 
> All of these problems can be solved, and I think solved better, by securing 
> the connection between the proxy and the back-end. That way the back end will 
> be able look not only for a specific header, but verify that the header came 
> from the proxy itself. An obscure header name is one way to do that, but it 
> has a lot of issues with it, especially since it’s likely to be selected once 
> during set-up and never changed throughout the lifetime of the deployment. I 
> think there are likely much better solutions here, and they’d address this 
> issue without things getting weird.
> 
> And one of the best things about a standard is that you’re still free to 
> completely ignore it if you want to, so people can and will keep following 
> whatever proprietary patterns they want to. But at least a standard mechanism 
> would give us a way to say to newcomers and veterans alike “No really, here’s 
> a way that we all agree works and has these properties”. 
> 
>  — Justin
> 
>> On Oct 30, 2019, at 3:43 AM, Neil Madden  wrote:
>> 
>> Replies below.
>> 
>>> On 29 Oct 2019, at 19:13, Justin Richer  wrote:
>>> 
>>> I would argue that making this standard would actually increase the 
>>> likelihood of developers getting this right, as now instead of following 
>>> some copy-pasted recipe for NGINX or Apache that they found on the web, 
>>> they could turn on a standard setting that would take care of both 
>>> stripping out incoming headers and injecting the appropriate values. And 
>>> all of that can be covered in the security considerations with a bunch of 
>>> normative text on top to make sure inbound headers are stripped. What 
>>> you’re describing below is clever, but ultimately it’s just a small bit of 
>>> obscurity more than anything real. 
>> 
>> Not really. If the header is cryptographically unguessable (eg includes a 
>> 128-bit base32-encoded string) then it provides a real security improvement. 
>> The header name becomes a bearer token effectively. There are many ways that 
>> a reverse proxy can be misconfigured in a way that compromises the security 
>> of trusted headers:
>> 
>> 1. There is the simple misconfiguration where you fail to strip trusted 
>> headers from incoming requests. For example in HAProxy this is the 
>> difference between set-header and add-header directives (and the add-header 
>> docs list passing a client cert to the backend as an example use case [1] 
>> despite this being insecure). Naive functional testing will still pass, but 
>> fairly basic adversarial tests would catch this. A standard header/config 
>> option might help with this simple case. 
>> 
>> 2. Scoping issues cause rules to not be applied where you expect them to. 
>> For example, nginx has multiple levels of scope that you can define header 
>> rules (http, server, location). But if you define *any* proxy_set_header 
>> directive at the location level (for example) it will ignore *all* such 
>> directives at the server or http levels, which can cause header-stripping 
>> rules to be silently disabled. 
>> 
>> 3. More advanced attacks exploit differences in how individual reverse 
>> proxies and application servers parse headers to smuggle headers or even 
>> whole requests past the RP [2]. 
>> 
>> Using unguessable header names for transmitting security-critical 
>> information between the RP and the app server provides an effective defense 
>> in depth against all of these attacks. (They are all forms of confused 
>> deputy attack and the unguessable header acts in the same way as an 
>> anti-CSRF token to prevent these systematically). 
>> 
>> (I had thought the random header name pattern was widely known, as I’ve 
>> heard it mentioned in conversations several times. But now I come to look 
>> for a definitive reference to it I can only find it mentioned in our own 
>> docs [3] and connect2id’s [4]). 
>> 
>> [1]: 
>> http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header
>> [2]: https://portswigger.net/web-security/request-smuggling
>> [3]: 
>> https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs
>> [4]: https://connect2id.com/products/server/docs/guides/tls-proxy 
>> 
>>> 
>>> The way things are today, you’ve got to not only pick a header and figure 
>>> out its format, but also do the injection protection step yourself. Since 
>>> all of these are disconnected, there are a lot more places that it could 
>>> fall over. Even a typo where you throw out incoming “CLIENT_CERT” but 
>>> inject “CLIENT_CERTS” or something like that would be disastrous. 
>> 
>> Most load balancers and reverse proxies support some way of setting headers 
>> that also strips the incoming header in one directive. Unless the standard 
>> gets rid of all the other ways they support to configure this (and removes 
>> all the old Stack Overflow recipes) then it won’t necessarily improve 
>> things. 
>> 
>> A

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Justin Richer]

All of these problems can be solved, and I think solved better, by securing the 
connection between the proxy and the back-end. That way the back end will be 
able look not only for a specific header, but verify that the header came from 
the proxy itself. An obscure header name is one way to do that, but it has a 
lot of issues with it, especially since it’s likely to be selected once during 
set-up and never changed throughout the lifetime of the deployment. I think 
there are likely much better solutions here, and they’d address this issue 
without things getting weird.

And one of the best things about a standard is that you’re still free to 
completely ignore it if you want to, so people can and will keep following 
whatever proprietary patterns they want to. But at least a standard mechanism 
would give us a way to say to newcomers and veterans alike “No really, here’s a 
way that we all agree works and has these properties”. 

 — Justin

> On Oct 30, 2019, at 3:43 AM, Neil Madden  wrote:
> 
> Replies below.
> 
>> On 29 Oct 2019, at 19:13, Justin Richer  wrote:
>> 
>> I would argue that making this standard would actually increase the 
>> likelihood of developers getting this right, as now instead of following 
>> some copy-pasted recipe for NGINX or Apache that they found on the web, they 
>> could turn on a standard setting that would take care of both stripping out 
>> incoming headers and injecting the appropriate values. And all of that can 
>> be covered in the security considerations with a bunch of normative text on 
>> top to make sure inbound headers are stripped. What you’re describing below 
>> is clever, but ultimately it’s just a small bit of obscurity more than 
>> anything real. 
> 
> Not really. If the header is cryptographically unguessable (eg includes a 
> 128-bit base32-encoded string) then it provides a real security improvement. 
> The header name becomes a bearer token effectively. There are many ways that 
> a reverse proxy can be misconfigured in a way that compromises the security 
> of trusted headers:
> 
> 1. There is the simple misconfiguration where you fail to strip trusted 
> headers from incoming requests. For example in HAProxy this is the difference 
> between set-header and add-header directives (and the add-header docs list 
> passing a client cert to the backend as an example use case [1] despite this 
> being insecure). Naive functional testing will still pass, but fairly basic 
> adversarial tests would catch this. A standard header/config option might 
> help with this simple case. 
> 
> 2. Scoping issues cause rules to not be applied where you expect them to. For 
> example, nginx has multiple levels of scope that you can define header rules 
> (http, server, location). But if you define *any* proxy_set_header directive 
> at the location level (for example) it will ignore *all* such directives at 
> the server or http levels, which can cause header-stripping rules to be 
> silently disabled. 
> 
> 3. More advanced attacks exploit differences in how individual reverse 
> proxies and application servers parse headers to smuggle headers or even 
> whole requests past the RP [2]. 
> 
> Using unguessable header names for transmitting security-critical information 
> between the RP and the app server provides an effective defense in depth 
> against all of these attacks. (They are all forms of confused deputy attack 
> and the unguessable header acts in the same way as an anti-CSRF token to 
> prevent these systematically). 
> 
> (I had thought the random header name pattern was widely known, as I’ve heard 
> it mentioned in conversations several times. But now I come to look for a 
> definitive reference to it I can only find it mentioned in our own docs [3] 
> and connect2id’s [4]). 
> 
> [1]: 
> http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header
>  
> 
> [2]: https://portswigger.net/web-security/request-smuggling 
> 
> [3]: 
> https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs 
> [4]:
>  https://connect2id.com/products/server/docs/guides/tls-proxy 
>  
> 
>> 
>> The way things are today, you’ve got to not only pick a header and figure 
>> out its format, but also do the injection protection step yourself. Since 
>> all of these are disconnected, there are a lot more places that it could 
>> fall over. Even a typo where you throw out incoming “CLIENT_CERT” but inject 
>> “CLIENT_CERTS” or something like that would be disastrous. 
> 
> Most load balancers and reverse proxies support some way of setting headers 
> that also strips the incoming header in one directive. Unless the standard 
> gets rid of all the othe

Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Neil Madden]

Replies below.

> On 29 Oct 2019, at 19:13, Justin Richer  wrote:
> 
> I would argue that making this standard would actually increase the 
> likelihood of developers getting this right, as now instead of following some 
> copy-pasted recipe for NGINX or Apache that they found on the web, they could 
> turn on a standard setting that would take care of both stripping out 
> incoming headers and injecting the appropriate values. And all of that can be 
> covered in the security considerations with a bunch of normative text on top 
> to make sure inbound headers are stripped. What you’re describing below is 
> clever, but ultimately it’s just a small bit of obscurity more than anything 
> real. 

Not really. If the header is cryptographically unguessable (eg includes a 
128-bit base32-encoded string) then it provides a real security improvement. 
The header name becomes a bearer token effectively. There are many ways that a 
reverse proxy can be misconfigured in a way that compromises the security of 
trusted headers:

1. There is the simple misconfiguration where you fail to strip trusted headers 
from incoming requests. For example in HAProxy this is the difference between 
set-header and add-header directives (and the add-header docs list passing a 
client cert to the backend as an example use case [1] despite this being 
insecure). Naive functional testing will still pass, but fairly basic 
adversarial tests would catch this. A standard header/config option might help 
with this simple case. 

2. Scoping issues cause rules to not be applied where you expect them to. For 
example, nginx has multiple levels of scope that you can define header rules 
(http, server, location). But if you define *any* proxy_set_header directive at 
the location level (for example) it will ignore *all* such directives at the 
server or http levels, which can cause header-stripping rules to be silently 
disabled. 

3. More advanced attacks exploit differences in how individual reverse proxies 
and application servers parse headers to smuggle headers or even whole requests 
past the RP [2]. 

Using unguessable header names for transmitting security-critical information 
between the RP and the app server provides an effective defense in depth 
against all of these attacks. (They are all forms of confused deputy attack and 
the unguessable header acts in the same way as an anti-CSRF token to prevent 
these systematically). 

(I had thought the random header name pattern was widely known, as I’ve heard 
it mentioned in conversations several times. But now I come to look for a 
definitive reference to it I can only find it mentioned in our own docs [3] and 
connect2id’s [4]). 

[1]: 
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header
[2]: https://portswigger.net/web-security/request-smuggling
[3]: 
https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs
[4]: https://connect2id.com/products/server/docs/guides/tls-proxy 

> 
> The way things are today, you’ve got to not only pick a header and figure out 
> its format, but also do the injection protection step yourself. Since all of 
> these are disconnected, there are a lot more places that it could fall over. 
> Even a typo where you throw out incoming “CLIENT_CERT” but inject 
> “CLIENT_CERTS” or something like that would be disastrous. 

Most load balancers and reverse proxies support some way of setting headers 
that also strips the incoming header in one directive. Unless the standard gets 
rid of all the other ways they support to configure this (and removes all the 
old Stack Overflow recipes) then it won’t necessarily improve things. 

A standard that provides an explicit field for including a secret (heck, we 
could even call it an access token :-) that SHOULD be used might provide a 
significant improvement in security. 

> 
> All in all, I am in favor of this being defined in one standard way, in 
> addition to secure communication between proxies and backends being 
> standardized — but this latter bit really seems like a separate problem.

I agree that that is a separate issue. 

— Neil

> 
>  — Justin
> 
>>> On Oct 28, 2019, at 12:32 PM, Neil Madden  wrote:
>>> 
>>> While there are some benefits to standardizing headers for this kind of 
>>> communication, there are some significant downsides - particularly when 
>>> using headers to communicate critical security information like certs. It 
>>> is *very* easy to misconfigure a reverse proxy to not strip Forwarded (or 
>>> whatever) headers from incoming requests, allowing a client to simply 
>>> supply a certificate as a header without authenticating the TLS connection 
>>> with the corresponding private key. One good practice to prevent this is to 
>>> pick a random and unguessable header name (configurable per installation) 
>>> to be used for communicating the certificate, rather than using something 
>>> fixed and standard. That way even if you misconfigur