Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Brian Bouterse]

tl;dr I'm going back to the original plan to have Content Guard
authorization in Pulp's processes and not the webserver. Also now we know
a) to use urlencoded certificates instead of stripping newlines and b) how
to make it work on both apache and nginx having tested it already.

After lots of testing and digging, I believe the best thing to do is to go
back to the original idea of having nginx/apache forward a certificate used
in a TLS connection with the client to pulp-container for authorization.
Here's the summary of reasons why:

* Trying to use a WSGIAccessScript to check in Python in the webserver
isn't an option when you're reverse proxying with either Apache or Nginx.
For example, to even do it in Nginx you would have to write the script in
Lua. Yikes. So moving the checking to apache/nginx isn't even possible in a
reasonable way.
* Having the additional requirement to install all of Pulp on nginx or
apache is undesirable at best. It's a requirement likely not easily
achieved in deployments like k8s for example meaning maybe Pulp couldn't
run there if we did this.
* Forward the certificate is not as bad as I originally thought. Yes you
are forwarding the clients cert but you aren't forwarding or re-presenting
the client's key so you can't impersonate the client like a MITM would.
Also you are forwarding it to a trusted service so that is ok.
* stripping newlines from rhsm certs causes them to be unreadable by rhsm
library so we need to use urlencoding instead of newline strippping to make
rhsm certs transport safe for headers
* urlencoding a tls cert for forwarding is so common nginx even has a
pre-defined variable for exactly this. For apache I have a simple
mod_rewrite rule handling it. Both were working for me.

Feedback is welcome! Please send any thoughts, questions, or concerns you
have. Next steps wise I will close the Epic I opened
https://pulp.plan.io/issues/6323 and it's subtasks as WONTFIX. I will
continue the pulp-certguard work on https://pulp.plan.io/issues/4664

-Brian




On Thu, Mar 12, 2020 at 9:44 AM Brian Bouterse  wrote:

>
>
> On Wed, Mar 11, 2020 at 9:18 PM Justin Sherrill 
> wrote:
>
>> Thanks and it does appears that we use certs that could be too large for
>> default header sizes (by several multiples).
>>
>> Could you elaborate a bit about the design a bit more?  I'm curious what
>> the requirements of web service layer (will it need to talk to the pulp3
>> api? the db?)  Will it just add some header after reading the cert (and
>> validating the path) and then pass it on to the reverse proxy with apache?
>>
> This was the idea, but in looking through the nginx and apache
> documentation it doesn't seem possible. I had planned to use
> WSGIAccessScript, but the pulp-content app isn't wsgi it's ProxyPass so I
> don't see an access script option for that which means my plan isn't
> achievable as is. At best nginx has the auth request model
> https://nginx.org/en/docs/http/ngx_http_auth_request_module.html but that
> makes a subrequest and apache doesn't support it. We'd still have the
> "certs are too big for headers" problem to deal with. I'm configuring nginx
> and http and doing some testing. Articles are saying the header options
> would work for both nginx and httpd, so I'm going to do some testing with
> their configs.
> https://connect2id.com/products/server/docs/guides/tls-proxy
>
> I'll post my findings and bump this list also for more input.
>
>> Thanks!
>>
>> Justin
>> On 3/11/20 2:58 PM, Brian Bouterse wrote:
>>
>>
>>
>> On Wed, Mar 11, 2020 at 2:34 PM Justin Sherrill 
>> wrote:
>>
>>> We had discussed base64 encoding the cert in the webserver on the way in
>>> and then letting cert guard decode it.  While that's not ideal I think it
>>> has some advantages over moving the full auth into the webserver.  What was
>>> your motivation for going with that approach over the base64 encoding
>>> approach?
>>>
>> Thank you for this question! I ended up with a few different concerns
>> about the base64 encode-and-forward idea. Architecturally the concern with
>> it is that it's frowned upon to forward the client's TLS cert beyond the
>> TLS termination point because that is what MITM software does. Also, there
>> are some practical concerns: one, I don't think nginx can provide a similar
>> runtime base64 encoding feature. Also I was concerned with header length
>> truncation and what happens when the certificates get longer.
>>
>> Overall having auth that is based on TLS certificates brought me to the
>> conclusion that we need to auth where the TLS is terminated. What do you
>> think?
>>
>> More thoughts and questions are welcome. This is a good discussion.
>>
>> On 3/11/20 2:11 PM, Brian Bouterse wrote:
>>>
>>> tl;dr: What we have today cannot work with rhsm certificates which
>>> Katello uses. To resolve, we need to have content guard checking moved to
>>> the webserver configs for apache and nginx and not done in pulp-content as
>>> it is today.  

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Brian Bouterse]

On Wed, Mar 11, 2020 at 9:18 PM Justin Sherrill  wrote:

> Thanks and it does appears that we use certs that could be too large for
> default header sizes (by several multiples).
>
> Could you elaborate a bit about the design a bit more?  I'm curious what
> the requirements of web service layer (will it need to talk to the pulp3
> api? the db?)  Will it just add some header after reading the cert (and
> validating the path) and then pass it on to the reverse proxy with apache?
>
This was the idea, but in looking through the nginx and apache
documentation it doesn't seem possible. I had planned to use
WSGIAccessScript, but the pulp-content app isn't wsgi it's ProxyPass so I
don't see an access script option for that which means my plan isn't
achievable as is. At best nginx has the auth request model
https://nginx.org/en/docs/http/ngx_http_auth_request_module.html but that
makes a subrequest and apache doesn't support it. We'd still have the
"certs are too big for headers" problem to deal with. I'm configuring nginx
and http and doing some testing. Articles are saying the header options
would work for both nginx and httpd, so I'm going to do some testing with
their configs. https://connect2id.com/products/server/docs/guides/tls-proxy

I'll post my findings and bump this list also for more input.

> Thanks!
>
> Justin
> On 3/11/20 2:58 PM, Brian Bouterse wrote:
>
>
>
> On Wed, Mar 11, 2020 at 2:34 PM Justin Sherrill 
> wrote:
>
>> We had discussed base64 encoding the cert in the webserver on the way in
>> and then letting cert guard decode it.  While that's not ideal I think it
>> has some advantages over moving the full auth into the webserver.  What was
>> your motivation for going with that approach over the base64 encoding
>> approach?
>>
> Thank you for this question! I ended up with a few different concerns
> about the base64 encode-and-forward idea. Architecturally the concern with
> it is that it's frowned upon to forward the client's TLS cert beyond the
> TLS termination point because that is what MITM software does. Also, there
> are some practical concerns: one, I don't think nginx can provide a similar
> runtime base64 encoding feature. Also I was concerned with header length
> truncation and what happens when the certificates get longer.
>
> Overall having auth that is based on TLS certificates brought me to the
> conclusion that we need to auth where the TLS is terminated. What do you
> think?
>
> More thoughts and questions are welcome. This is a good discussion.
>
> On 3/11/20 2:11 PM, Brian Bouterse wrote:
>>
>> tl;dr: What we have today cannot work with rhsm certificates which
>> Katello uses. To resolve, we need to have content guard checking moved to
>> the webserver configs for apache and nginx and not done in pulp-content as
>> it is today.  https://pulp.plan.io/issues/6323
>>
>> We need to bring the auth to where TLS is terminated because we can't
>> being the client certs to pulp-content due to invalid header characters. As
>> is, pulp-certguard cannot work with Katello's cert types (rhsm certs) so
>> that is driving my changes.
>>
>> If anyone has major concerns or other ideas please let me know. In the
>> meantime I'm proceeding moving the authorization to the webserver and then
>> updating pulp-certguard to work with that. This will make pulp-certguard's
>> GA tied to pulpcore 3.3.0. Feedback is welcome.
>>
>> [0]: https://pulp.plan.io/issues/6323
>>
>> Thanks,
>> Brian
>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Justin Sherrill]

Thanks and it does appears that we use certs that could be too large for 
default header sizes (by several multiples).


Could you elaborate a bit about the design a bit more?  I'm curious what 
the requirements of web service layer (will it need to talk to the pulp3 
api? the db?)  Will it just add some header after reading the cert (and 
validating the path) and then pass it on to the reverse proxy with apache?


Thanks!

Justin

On 3/11/20 2:58 PM, Brian Bouterse wrote:



On Wed, Mar 11, 2020 at 2:34 PM Justin Sherrill > wrote:


We had discussed base64 encoding the cert in the webserver on the
way in and then letting cert guard decode it.  While that's not
ideal I think it has some advantages over moving the full auth
into the webserver.  What was your motivation for going with that
approach over the base64 encoding approach?

Thank you for this question! I ended up with a few different concerns 
about the base64 encode-and-forward idea. Architecturally the concern 
with it is that it's frowned upon to forward the client's TLS cert 
beyond the TLS termination point because that is what MITM software 
does. Also, there are some practical concerns: one, I don't think 
nginx can provide a similar runtime base64 encoding feature. Also I 
was concerned with header length truncation and what happens when the 
certificates get longer.


Overall having auth that is based on TLS certificates brought me to 
the conclusion that we need to auth where the TLS is terminated. What 
do you think?


More thoughts and questions are welcome. This is a good discussion.

On 3/11/20 2:11 PM, Brian Bouterse wrote:

tl;dr: What we have today cannot work with rhsm certificates
which Katello uses. To resolve, we need to have content guard
checking moved to the webserver configs for apache and nginx and
not done in pulp-content as it is today.
https://pulp.plan.io/issues/6323

We need to bring the auth to where TLS is terminated because we
can't being the client certs to pulp-content due to invalid
header characters. As is, pulp-certguard cannot work with
Katello's cert types (rhsm certs) so that is driving my changes.

If anyone has major concerns or other ideas please let me know.
In the meantime I'm proceeding moving the authorization to the
webserver and then updating pulp-certguard to work with that.
This will make pulp-certguard's GA tied to pulpcore 3.3.0.
Feedback is welcome.

[0]: https://pulp.plan.io/issues/6323

Thanks,
Brian


___
Pulp-dev mailing list
Pulp-dev@redhat.com  
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[James Cassell]

On Wed, Mar 11, 2020, at 2:58 PM, Brian Bouterse wrote:
> 
> 
> On Wed, Mar 11, 2020 at 2:34 PM Justin Sherrill  wrote:
> > We had discussed base64 encoding the cert in the webserver on the way in 
> > and then letting cert guard decode it. While that's not ideal I think it 
> > has some advantages over moving the full auth into the webserver. What was 
> > your motivation for going with that approach over the base64 encoding 
> > approach? 
> 
> Thank you for this question! I ended up with a few different concerns 
> about the base64 encode-and-forward idea. Architecturally the concern 
> with it is that it's frowned upon to forward the client's TLS cert 
> beyond the TLS termination point because that is what MITM software 
> does. Also, there are some practical concerns: one, I don't think nginx 
> can provide a similar runtime base64 encoding feature. Also I was 
> concerned with header length truncation and what happens when the 
> certificates get longer.
> 
> Overall having auth that is based on TLS certificates brought me to the 
> conclusion that we need to auth where the TLS is terminated. What do 
> you think?
> 
> More thoughts and questions are welcome. This is a good discussion.
> 

I agree with your points.

Also, having the auth in the web server would allow me (random bystander) to 
use the functionality without actually running a full pulp; just plain 
candlepin with yum repos behind a web server.


V/r,
James Cassell

> 
> > On 3/11/20 2:11 PM, Brian Bouterse wrote:
> >> tl;dr: What we have today cannot work with rhsm certificates which Katello 
> >> uses. To resolve, we need to have content guard checking moved to the 
> >> webserver configs for apache and nginx and not done in pulp-content as it 
> >> is today. https://pulp.plan.io/issues/6323
> >> 
> >> We need to bring the auth to where TLS is terminated because we can't 
> >> being the client certs to pulp-content due to invalid header characters. 
> >> As is, pulp-certguard cannot work with Katello's cert types (rhsm certs) 
> >> so that is driving my changes.
> >> 
> >> If anyone has major concerns or other ideas please let me know. In the 
> >> meantime I'm proceeding moving the authorization to the webserver and then 
> >> updating pulp-certguard to work with that. This will make pulp-certguard's 
> >> GA tied to pulpcore 3.3.0. Feedback is welcome.
> >> 
> >> [0]: https://pulp.plan.io/issues/6323
> >> 
> >> Thanks,
> >> Brian
> >> 


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Brian Bouterse]

On Wed, Mar 11, 2020 at 2:34 PM Justin Sherrill  wrote:

> We had discussed base64 encoding the cert in the webserver on the way in
> and then letting cert guard decode it.  While that's not ideal I think it
> has some advantages over moving the full auth into the webserver.  What was
> your motivation for going with that approach over the base64 encoding
> approach?
>
Thank you for this question! I ended up with a few different concerns about
the base64 encode-and-forward idea. Architecturally the concern with it is
that it's frowned upon to forward the client's TLS cert beyond the TLS
termination point because that is what MITM software does. Also, there are
some practical concerns: one, I don't think nginx can provide a similar
runtime base64 encoding feature. Also I was concerned with header length
truncation and what happens when the certificates get longer.

Overall having auth that is based on TLS certificates brought me to the
conclusion that we need to auth where the TLS is terminated. What do you
think?

More thoughts and questions are welcome. This is a good discussion.

On 3/11/20 2:11 PM, Brian Bouterse wrote:
>
> tl;dr: What we have today cannot work with rhsm certificates which Katello
> uses. To resolve, we need to have content guard checking moved to the
> webserver configs for apache and nginx and not done in pulp-content as it
> is today.  https://pulp.plan.io/issues/6323
>
> We need to bring the auth to where TLS is terminated because we can't
> being the client certs to pulp-content due to invalid header characters. As
> is, pulp-certguard cannot work with Katello's cert types (rhsm certs) so
> that is driving my changes.
>
> If anyone has major concerns or other ideas please let me know. In the
> meantime I'm proceeding moving the authorization to the webserver and then
> updating pulp-certguard to work with that. This will make pulp-certguard's
> GA tied to pulpcore 3.3.0. Feedback is welcome.
>
> [0]: https://pulp.plan.io/issues/6323
>
> Thanks,
> Brian
>
>
> ___
> Pulp-dev mailing 
> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Justin Sherrill]

We had discussed base64 encoding the cert in the webserver on the way in 
and then letting cert guard decode it.  While that's not ideal I think 
it has some advantages over moving the full auth into the webserver.  
What was your motivation for going with that approach over the base64 
encoding approach?


On 3/11/20 2:11 PM, Brian Bouterse wrote:
tl;dr: What we have today cannot work with rhsm certificates which 
Katello uses. To resolve, we need to have content guard checking moved 
to the webserver configs for apache and nginx and not done in 
pulp-content as it is today. https://pulp.plan.io/issues/6323


We need to bring the auth to where TLS is terminated because we can't 
being the client certs to pulp-content due to invalid header 
characters. As is, pulp-certguard cannot work with Katello's cert 
types (rhsm certs) so that is driving my changes.


If anyone has major concerns or other ideas please let me know. In the 
meantime I'm proceeding moving the authorization to the webserver and 
then updating pulp-certguard to work with that. This will make 
pulp-certguard's GA tied to pulpcore 3.3.0. Feedback is welcome.


[0]: https://pulp.plan.io/issues/6323

Thanks,
Brian


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Brian Bouterse]

On Wed, Mar 11, 2020 at 2:18 PM Eric Helms  wrote:

>
>
> On Wed, Mar 11, 2020 at 2:12 PM Brian Bouterse 
> wrote:
>
>> tl;dr: What we have today cannot work with rhsm certificates which
>> Katello uses. To resolve, we need to have content guard checking moved to
>> the webserver configs for apache and nginx and not done in pulp-content as
>> it is today.  https://pulp.plan.io/issues/6323
>>
>> We need to bring the auth to where TLS is terminated because we can't
>> being the client certs to pulp-content due to invalid header characters. As
>> is, pulp-certguard cannot work with Katello's cert types (rhsm certs) so
>> that is driving my changes.
>>
>> If anyone has major concerns or other ideas please let me know. In the
>> meantime I'm proceeding moving the authorization to the webserver and then
>> updating pulp-certguard to work with that. This will make pulp-certguard's
>> GA tied to pulpcore 3.3.0. Feedback is welcome.
>>
>
> What will this mean from a runtime perspective? Or rather, what within the
> webserver layer will be handling this auth?
>
The plan is that it will be a python "access script" the same way we did it
in pulp2. The ansible-pulp installer will configure Nginx or Apache to use
the access scripts on all installations starting with 3.3. For installs
that have another installer, e.g. Katello, the same apache config the Pulp
installer installs can be used in Katello's installer.

What do you think about this? Is this what you were asking?


>
>>
>> [0]: https://pulp.plan.io/issues/6323
>>
>> Thanks,
>> Brian
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
> Eric Helms
> Principal Software Engineer
> Satellite and Cloud Services
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Eric Helms]

On Wed, Mar 11, 2020 at 2:12 PM Brian Bouterse  wrote:

> tl;dr: What we have today cannot work with rhsm certificates which Katello
> uses. To resolve, we need to have content guard checking moved to the
> webserver configs for apache and nginx and not done in pulp-content as it
> is today.  https://pulp.plan.io/issues/6323
>
> We need to bring the auth to where TLS is terminated because we can't
> being the client certs to pulp-content due to invalid header characters. As
> is, pulp-certguard cannot work with Katello's cert types (rhsm certs) so
> that is driving my changes.
>
> If anyone has major concerns or other ideas please let me know. In the
> meantime I'm proceeding moving the authorization to the webserver and then
> updating pulp-certguard to work with that. This will make pulp-certguard's
> GA tied to pulpcore 3.3.0. Feedback is welcome.
>

What will this mean from a runtime perspective? Or rather, what within the
webserver layer will be handling this auth?


>
> [0]: https://pulp.plan.io/issues/6323
>
> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>


-- 
Eric Helms
Principal Software Engineer
Satellite and Cloud Services
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Brian Bouterse]

tl;dr: What we have today cannot work with rhsm certificates which Katello
uses. To resolve, we need to have content guard checking moved to the
webserver configs for apache and nginx and not done in pulp-content as it
is today.  https://pulp.plan.io/issues/6323

We need to bring the auth to where TLS is terminated because we can't being
the client certs to pulp-content due to invalid header characters. As is,
pulp-certguard cannot work with Katello's cert types (rhsm certs) so that
is driving my changes.

If anyone has major concerns or other ideas please let me know. In the
meantime I'm proceeding moving the authorization to the webserver and then
updating pulp-certguard to work with that. This will make pulp-certguard's
GA tied to pulpcore 3.3.0. Feedback is welcome.

[0]: https://pulp.plan.io/issues/6323

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev