Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Mon, 18 Nov 2019 at 13:05, Clayton Coleman  wrote:

> Raise a bug to the installler component, yes
>

Ok thanks, I raised a bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1773419


> On Nov 17, 2019, at 6:03 PM, Joel Pearson 
> wrote:
>
> On Mon, 18 Nov 2019 at 12:37, Ben Parees  wrote:
>
>>
>>
>> On Sun, Nov 17, 2019 at 7:24 PM Joel Pearson <
>> japear...@agiledigital.com.au> wrote:
>>
>>>
>>>
>>> On Wed, 13 Nov 2019 at 02:43, Ben Parees  wrote:
>>>


 On Mon, Nov 11, 2019 at 11:27 PM Ben Parees  wrote:

>
>
> On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>>
>>
>> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>>
>>>
>>>

 Can I use the “trustedCA” part of the proxy configuration without
 actually specifying an explicit proxy?

>>>
>>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>>> consider it a bug).
>>>
>>> It does work! Thanks for that. user-ca-bundle already existed and
>> had my certificate in there, I just needed to reference user-ca-bundle in
>> the proxy config.
>>
>
> cool, given that you supplied the CAs during install, and the
> user-ca-bundle CM was created, i'm a little surprised the install didn't
> automatically setup the reference in the proxyconfig resource for you.  
> I'm
> guessing it did not because there was no actual proxy hostname configured.
> I think that's a gap we should close..would you mind filing a bug?  (
> bugzilla.redhat.com).  You can submit it against the install
> component.
>

 fyi I've filed a bug for this aspect of the issues you ran into:
 https://bugzilla.redhat.com/show_bug.cgi?id=1771564


>>> Thanks for raising this, reading through the related github tickets it
>>> looks like I've opened a can of worms to some degree.
>>>
>>
>> Yes there's some difference of opinion on what the out of box desired
>> behavior is, but at a minimum you've exposed a gap in our documentation
>> that we will get fixed.
>>
>>
>> I also just discovered that the openshift cluster version operator (CVO),
> isn't quite configured correctly out of the box to use the correct trusted
> CA certs (which means it can't download cluster updates).
>
> It correctly mounts /etc/ssl/certs from the host (the masters), but it
> fails to also mount /etc/pki, because the certs are a symlink
> /etc/ssl/certs/ca-bundle.crt ->
> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
>
> I couldn't find where the installer sets up the CVO but an example of what
> is missing is here.
>
> https://github.com/openshift/cluster-version-operator/blob/01a7825179246fa708ac64de96e6675c0bf9a930/bootstrap/bootstrap-pod.yaml#L44-L46
>
>
> Is there an existing bug for this? Or should I raise a bugzilla for this?
> Would it be part of the installer?
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Clayton Coleman]

Raise a bug to the installler component, yes

On Nov 17, 2019, at 6:03 PM, Joel Pearson 
wrote:

On Mon, 18 Nov 2019 at 12:37, Ben Parees  wrote:

>
>
> On Sun, Nov 17, 2019 at 7:24 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>>
>>
>> On Wed, 13 Nov 2019 at 02:43, Ben Parees  wrote:
>>
>>>
>>>
>>> On Mon, Nov 11, 2019 at 11:27 PM Ben Parees  wrote:
>>>


 On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson <
 japear...@agiledigital.com.au> wrote:

>
>
> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>
>>
>>
>>>
>>> Can I use the “trustedCA” part of the proxy configuration without
>>> actually specifying an explicit proxy?
>>>
>>
>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>> consider it a bug).
>>
>> It does work! Thanks for that. user-ca-bundle already existed and had
> my certificate in there, I just needed to reference user-ca-bundle in the
> proxy config.
>

 cool, given that you supplied the CAs during install, and the
 user-ca-bundle CM was created, i'm a little surprised the install didn't
 automatically setup the reference in the proxyconfig resource for you.  I'm
 guessing it did not because there was no actual proxy hostname configured.
 I think that's a gap we should close..would you mind filing a bug?  (
 bugzilla.redhat.com).  You can submit it against the install component.

>>>
>>> fyi I've filed a bug for this aspect of the issues you ran into:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1771564
>>>
>>>
>> Thanks for raising this, reading through the related github tickets it
>> looks like I've opened a can of worms to some degree.
>>
>
> Yes there's some difference of opinion on what the out of box desired
> behavior is, but at a minimum you've exposed a gap in our documentation
> that we will get fixed.
>
>
> I also just discovered that the openshift cluster version operator (CVO),
isn't quite configured correctly out of the box to use the correct trusted
CA certs (which means it can't download cluster updates).

It correctly mounts /etc/ssl/certs from the host (the masters), but it
fails to also mount /etc/pki, because the certs are a symlink
/etc/ssl/certs/ca-bundle.crt ->
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

I couldn't find where the installer sets up the CVO but an example of what
is missing is here.
https://github.com/openshift/cluster-version-operator/blob/01a7825179246fa708ac64de96e6675c0bf9a930/bootstrap/bootstrap-pod.yaml#L44-L46


Is there an existing bug for this? Or should I raise a bugzilla for this?
Would it be part of the installer?

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Mon, 18 Nov 2019 at 12:37, Ben Parees  wrote:

>
>
> On Sun, Nov 17, 2019 at 7:24 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>>
>>
>> On Wed, 13 Nov 2019 at 02:43, Ben Parees  wrote:
>>
>>>
>>>
>>> On Mon, Nov 11, 2019 at 11:27 PM Ben Parees  wrote:
>>>


 On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson <
 japear...@agiledigital.com.au> wrote:

>
>
> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>
>>
>>
>>>
>>> Can I use the “trustedCA” part of the proxy configuration without
>>> actually specifying an explicit proxy?
>>>
>>
>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>> consider it a bug).
>>
>> It does work! Thanks for that. user-ca-bundle already existed and had
> my certificate in there, I just needed to reference user-ca-bundle in the
> proxy config.
>

 cool, given that you supplied the CAs during install, and the
 user-ca-bundle CM was created, i'm a little surprised the install didn't
 automatically setup the reference in the proxyconfig resource for you.  I'm
 guessing it did not because there was no actual proxy hostname configured.
 I think that's a gap we should close..would you mind filing a bug?  (
 bugzilla.redhat.com).  You can submit it against the install component.

>>>
>>> fyi I've filed a bug for this aspect of the issues you ran into:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1771564
>>>
>>>
>> Thanks for raising this, reading through the related github tickets it
>> looks like I've opened a can of worms to some degree.
>>
>
> Yes there's some difference of opinion on what the out of box desired
> behavior is, but at a minimum you've exposed a gap in our documentation
> that we will get fixed.
>
>
> I also just discovered that the openshift cluster version operator (CVO),
isn't quite configured correctly out of the box to use the correct trusted
CA certs (which means it can't download cluster updates).

It correctly mounts /etc/ssl/certs from the host (the masters), but it
fails to also mount /etc/pki, because the certs are a symlink
/etc/ssl/certs/ca-bundle.crt ->
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

I couldn't find where the installer sets up the CVO but an example of what
is missing is here.
https://github.com/openshift/cluster-version-operator/blob/01a7825179246fa708ac64de96e6675c0bf9a930/bootstrap/bootstrap-pod.yaml#L44-L46


Is there an existing bug for this? Or should I raise a bugzilla for this?
Would it be part of the installer?
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Sun, Nov 17, 2019 at 7:24 PM Joel Pearson 
wrote:

>
>
> On Wed, 13 Nov 2019 at 02:43, Ben Parees  wrote:
>
>>
>>
>> On Mon, Nov 11, 2019 at 11:27 PM Ben Parees  wrote:
>>
>>>
>>>
>>> On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson <
>>> japear...@agiledigital.com.au> wrote:
>>>


 On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:

>
>
>>
>> Can I use the “trustedCA” part of the proxy configuration without
>> actually specifying an explicit proxy?
>>
>
> you should be able to.  Daneyon can you confirm?  (if you can't i'd
> consider it a bug).
>
> It does work! Thanks for that. user-ca-bundle already existed and had
 my certificate in there, I just needed to reference user-ca-bundle in the
 proxy config.

>>>
>>> cool, given that you supplied the CAs during install, and the
>>> user-ca-bundle CM was created, i'm a little surprised the install didn't
>>> automatically setup the reference in the proxyconfig resource for you.  I'm
>>> guessing it did not because there was no actual proxy hostname configured.
>>> I think that's a gap we should close..would you mind filing a bug?  (
>>> bugzilla.redhat.com).  You can submit it against the install component.
>>>
>>
>> fyi I've filed a bug for this aspect of the issues you ran into:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1771564
>>
>>
> Thanks for raising this, reading through the related github tickets it
> looks like I've opened a can of worms to some degree.
>

Yes there's some difference of opinion on what the out of box desired
behavior is, but at a minimum you've exposed a gap in our documentation
that we will get fixed.



-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Wed, 13 Nov 2019 at 01:34, Ben Parees  wrote:

>
>
> On Tue, Nov 12, 2019 at 3:45 AM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>>
>>
>> On Tue, 12 Nov 2019 at 15:37, Ben Parees  wrote:
>>
>>>
>>>
>>> On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson <
>>> japear...@agiledigital.com.au> wrote:
>>>
 I've now discovered that the cluster-samples-operator doesn't seem
 honour the proxy settings, and I see lots of errors in the
 cluster-samples-operator- pod logs

 time="2019-11-12T04:15:49Z" level=warning msg="Image import for
 imagestream dotnet tag 2.1 generation 2 failed with detailed message
 Internal error occurred: Get https://I /v2/
 : x509: certificate signed by unknown
 authority"

 Is there a way to get that operator to use the same user-ca-bundle?

>>>
>>> image import should be using those CAs (it's really about the
>>> openshift-apiserver, not the samples operator) automatically (sounds like
>>> another potential bug, but i'll let Oleg weigh in on this one).
>>>
>>> However barring that, you can use the mechanism described here to
>>> setup additional CAs for importing from registries:
>>>
>>> https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration
>>>
>>> you can follow the more detailed instructions here:
>>>
>>> https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca
>>>
>>
>> I tried this approach but it didn't work for me.
>>
>> I ran this command:
>>
>> oc create configmap registry-cas -n openshift-config \
>> --from-file=registry.redhat.io..5000=/path/to/ca.crt \
>> --from-file=registry.redhat.io..443=/path/to/ca.crt \
>> --from-file=registry.redhat.io=/path/to/ca.crt
>>
>> and:
>>
>> oc patch image.config.openshift.io/cluster --patch
>> '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
>>
>> And that still didn't work. First I deleted the
>> cluster-samples-operator- pod, then I tried forcing the masters to
>> restart by touching some machine config (I don't know a better way).
>> But it still didn't work.  Maybe the samples operator doesn't let you
>> easily override the trusted CA certs?
>>
>
> Because no good bug report should not be rewarded with some educational
> background:
>
> The samples operator is only responsible for creating the imagestream, it
> isn't actually doing the import (ie reaching out to the registry and
> pulling down the metadata and putting it in the imagestream).  That task is
> performed by the openshift-apiserver.  What should be happening when you
> update the image config resource with the name of the CA configmap is that
> the openshift-apiserver operator should observe the configuration change
> and provide the new CAs to the openshift-apiserver pods (which necessitates
> a restart of the openshift-apiserver pods).
>
> Once the openshift-apiserver pods are restarted with the new CAs, you
> should be able to run "oc import-image" to retry the import.  (The samples
> operator is supposed to retry the failed imports periodically, but there is
> a different bug that is being fixed related to that, so until then you'll
> have to retry the import manually once you've corrected whatever caused the
> failure).
>
> So again, there may be a bug here in terms of the openshift-apiserver
> picking up the CAs and we need to investigate it (as well as a separate bug
> if it is not picking up the proxy CAs), but I wanted you to understand the
> relevant components so your own debugging process can be more productive.
>
>
Thanks for the explanation Ben, it helped me figure out there the issues
where.

For other reasons (I had tried to customise name of the Azure
resource group and something would occasionally (once a day?) change it
back to the default name), I ended up burning the cluster to the ground,
and I configured the "spec.trustedCA.name" reference during installation by
customising the manifest files generated by "openshift-install create
manifests --dir=ignition-files" and then the samples operators worked out
of the box!

So it was just that the samples operator doesn't retry as you mentioned.

Also, I didn't need to setup the additional CAs for importing from
registries in the end.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Mon, Nov 11, 2019 at 11:27 PM Ben Parees  wrote:

>
>
> On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>>
>>
>> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>>
>>>
>>>

 Can I use the “trustedCA” part of the proxy configuration without
 actually specifying an explicit proxy?

>>>
>>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>>> consider it a bug).
>>>
>>> It does work! Thanks for that. user-ca-bundle already existed and had my
>> certificate in there, I just needed to reference user-ca-bundle in the
>> proxy config.
>>
>
> cool, given that you supplied the CAs during install, and the
> user-ca-bundle CM was created, i'm a little surprised the install didn't
> automatically setup the reference in the proxyconfig resource for you.  I'm
> guessing it did not because there was no actual proxy hostname configured.
> I think that's a gap we should close..would you mind filing a bug?  (
> bugzilla.redhat.com).  You can submit it against the install component.
>

fyi I've filed a bug for this aspect of the issues you ran into:
https://bugzilla.redhat.com/show_bug.cgi?id=1771564

we still need to chase down the issues you hit with respect to the various
CAs (the cluster proxy CA config and the image CA config) seemingly not
being used during image import, there are no tracker bugs for those yet but
Oleg is investigating.



>
>
>
>>
>> apiVersion: config.openshift.io/v1
>> kind: Proxy
>> metadata:
>>   name: cluster
>> spec:
>>   trustedCA:
>> name: user-ca-bundle
>>
>
>
> --
> Ben Parees | OpenShift
>
>

-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Gabe Montero]

On Mon, Nov 11, 2019 at 11:27 PM Joel Pearson 
wrote:

> I've now discovered that the cluster-samples-operator doesn't seem honour
> the proxy settings, and I see lots of errors in the
> cluster-samples-operator- pod logs
>
> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
> imagestream dotnet tag 2.1 generation 2 failed with detailed message
> Internal error occurred: Get https://registry.redhat.io/v2/: x509:
> certificate signed by unknown authority"
>
> Is there a way to get that operator to use the same user-ca-bundle?
>

Samples operator just reports the status of the sample imagestreams.  It
does not actually execute the imagestream import, and thus is not the
controller that consumes the user-ca-bundle.

Imagestream import is a function of the imagestream controller in the
openshift-controller-manager and the internal image registry.

That said, my understanding was those items should consume global CA /
cluster image configuration as well.

The folks on here already, plus Oleg, who I have now included, can
elaborate.  My quick scan of the docs did not find where that was explained.


> On Tue, 12 Nov 2019 at 14:46, Joel Pearson 
> wrote:
>
>>
>>
>> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>>
>>>
>>>

 Can I use the “trustedCA” part of the proxy configuration without
 actually specifying an explicit proxy?

>>>
>>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>>> consider it a bug).
>>>
>>> It does work! Thanks for that. user-ca-bundle already existed and had my
>> certificate in there, I just needed to reference user-ca-bundle in the
>> proxy config.
>>
>> apiVersion: config.openshift.io/v1
>> kind: Proxy
>> metadata:
>>   name: cluster
>> spec:
>>   trustedCA:
>> name: user-ca-bundle
>>
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Tue, Nov 12, 2019 at 3:45 AM Joel Pearson 
wrote:

>
>
> On Tue, 12 Nov 2019 at 15:37, Ben Parees  wrote:
>
>>
>>
>> On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson <
>> japear...@agiledigital.com.au> wrote:
>>
>>> I've now discovered that the cluster-samples-operator doesn't seem
>>> honour the proxy settings, and I see lots of errors in the
>>> cluster-samples-operator- pod logs
>>>
>>> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
>>> imagestream dotnet tag 2.1 generation 2 failed with detailed message
>>> Internal error occurred: Get https://I /v2/
>>> : x509: certificate signed by unknown
>>> authority"
>>>
>>> Is there a way to get that operator to use the same user-ca-bundle?
>>>
>>
>> image import should be using those CAs (it's really about the
>> openshift-apiserver, not the samples operator) automatically (sounds like
>> another potential bug, but i'll let Oleg weigh in on this one).
>>
>> However barring that, you can use the mechanism described here to
>> setup additional CAs for importing from registries:
>>
>> https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration
>>
>> you can follow the more detailed instructions here:
>>
>> https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca
>>
>
> I tried this approach but it didn't work for me.
>
> I ran this command:
>
> oc create configmap registry-cas -n openshift-config \
> --from-file=registry.redhat.io..5000=/path/to/ca.crt \
> --from-file=registry.redhat.io..443=/path/to/ca.crt \
> --from-file=registry.redhat.io=/path/to/ca.crt
>
> and:
>
> oc patch image.config.openshift.io/cluster --patch
> '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
>
> And that still didn't work. First I deleted the
> cluster-samples-operator- pod, then I tried forcing the masters to
> restart by touching some machine config (I don't know a better way).
> But it still didn't work.  Maybe the samples operator doesn't let you
> easily override the trusted CA certs?
>

Because no good bug report should not be rewarded with some educational
background:

The samples operator is only responsible for creating the imagestream, it
isn't actually doing the import (ie reaching out to the registry and
pulling down the metadata and putting it in the imagestream).  That task is
performed by the openshift-apiserver.  What should be happening when you
update the image config resource with the name of the CA configmap is that
the openshift-apiserver operator should observe the configuration change
and provide the new CAs to the openshift-apiserver pods (which necessitates
a restart of the openshift-apiserver pods).

Once the openshift-apiserver pods are restarted with the new CAs, you
should be able to run "oc import-image" to retry the import.  (The samples
operator is supposed to retry the failed imports periodically, but there is
a different bug that is being fixed related to that, so until then you'll
have to retry the import manually once you've corrected whatever caused the
failure).

So again, there may be a bug here in terms of the openshift-apiserver
picking up the CAs and we need to investigate it (as well as a separate bug
if it is not picking up the proxy CAs), but I wanted you to understand the
relevant components so your own debugging process can be more productive.




>
>
>>
>>
>> (Brandi/Adam, we should really include the example from that second link,
>> in the general "image resource configuration" page from the first link).
>>
>> Unfortunately it does not allow you to reuse the user-ca-bundle CM since
>> the format of the CM is a bit different (needs an entry per registry
>> hostname).
>>
>>

-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Adam Kaplan]

Slightly related - there is an existing bugzilla where `oc import-image`
and `oc tag` will fail if the "origin" tag references the internal registry
with a similar x509 error [1].
Echoing Clayton, please file a bug and if warranted we'll link the two
together.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1716835

On Tue, Nov 12, 2019 at 8:42 AM Clayton Coleman  wrote:

>
>
> On Nov 12, 2019, at 3:44 AM, Joel Pearson 
> wrote:
>
>
>
> On Tue, 12 Nov 2019 at 15:37, Ben Parees  wrote:
>
>>
>>
>> On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson <
>> japear...@agiledigital.com.au> wrote:
>>
>>> I've now discovered that the cluster-samples-operator doesn't seem
>>> honour the proxy settings, and I see lots of errors in the
>>> cluster-samples-operator- pod logs
>>>
>>> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
>>> imagestream dotnet tag 2.1 generation 2 failed with detailed message
>>> Internal error occurred: Get https://I /v2/
>>> : x509: certificate signed by unknown
>>> authority"
>>>
>>> Is there a way to get that operator to use the same user-ca-bundle?
>>>
>>
>> image import should be using those CAs (it's really about the
>> openshift-apiserver, not the samples operator) automatically (sounds like
>> another potential bug, but i'll let Oleg weigh in on this one).
>>
>> However barring that, you can use the mechanism described here to
>> setup additional CAs for importing from registries:
>>
>> https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration
>>
>> you can follow the more detailed instructions here:
>>
>> https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca
>>
>
> I tried this approach but it didn't work for me.
>
> I ran this command:
>
> oc create configmap registry-cas -n openshift-config \
> --from-file=registry.redhat.io..5000=/path/to/ca.crt \
> --from-file=registry.redhat.io..443=/path/to/ca.crt \
> --from-file=registry.redhat.io=/path/to/ca.crt
>
> and:
>
> oc patch image.config.openshift.io/cluster --patch
> '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
>
> And that still didn't work. First I deleted the
> cluster-samples-operator- pod, then I tried forcing the masters to
> restart by touching some machine config (I don't know a better way).
> But it still didn't work.  Maybe the samples operator doesn't let you
> easily override the trusted CA certs?
>
>
> No, as Ben said this should be working.  Please file a bug.
>
>
>
>>
>>
>> (Brandi/Adam, we should really include the example from that second link,
>> in the general "image resource configuration" page from the first link).
>>
>> Unfortunately it does not allow you to reuse the user-ca-bundle CM since
>> the format of the CM is a bit different (needs an entry per registry
>> hostname).
>>
>> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>

-- 

Adam Kaplan

He/Him

Senior Software Engineer - OpenShift

Red Hat 

100 E. Davie St. Raleigh, NC 27601 USA

adam.kap...@redhat.comT: +1-919-754-4843 IM: adambkaplan

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Clayton Coleman]

On Nov 12, 2019, at 3:44 AM, Joel Pearson 
wrote:



On Tue, 12 Nov 2019 at 15:37, Ben Parees  wrote:

>
>
> On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>> I've now discovered that the cluster-samples-operator doesn't seem honour
>> the proxy settings, and I see lots of errors in the
>> cluster-samples-operator- pod logs
>>
>> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
>> imagestream dotnet tag 2.1 generation 2 failed with detailed message
>> Internal error occurred: Get https://I /v2/
>> : x509: certificate signed by unknown
>> authority"
>>
>> Is there a way to get that operator to use the same user-ca-bundle?
>>
>
> image import should be using those CAs (it's really about the
> openshift-apiserver, not the samples operator) automatically (sounds like
> another potential bug, but i'll let Oleg weigh in on this one).
>
> However barring that, you can use the mechanism described here to
> setup additional CAs for importing from registries:
>
> https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration
>
> you can follow the more detailed instructions here:
>
> https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca
>

I tried this approach but it didn't work for me.

I ran this command:

oc create configmap registry-cas -n openshift-config \
--from-file=registry.redhat.io..5000=/path/to/ca.crt \
--from-file=registry.redhat.io..443=/path/to/ca.crt \
--from-file=registry.redhat.io=/path/to/ca.crt

and:

oc patch image.config.openshift.io/cluster --patch
'{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge

And that still didn't work. First I deleted the
cluster-samples-operator- pod, then I tried forcing the masters to
restart by touching some machine config (I don't know a better way).
But it still didn't work.  Maybe the samples operator doesn't let you
easily override the trusted CA certs?


No, as Ben said this should be working.  Please file a bug.



>
>
> (Brandi/Adam, we should really include the example from that second link,
> in the general "image resource configuration" page from the first link).
>
> Unfortunately it does not allow you to reuse the user-ca-bundle CM since
> the format of the CM is a bit different (needs an entry per registry
> hostname).
>
> ___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Tue, 12 Nov 2019 at 15:37, Ben Parees  wrote:

>
>
> On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>> I've now discovered that the cluster-samples-operator doesn't seem honour
>> the proxy settings, and I see lots of errors in the
>> cluster-samples-operator- pod logs
>>
>> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
>> imagestream dotnet tag 2.1 generation 2 failed with detailed message
>> Internal error occurred: Get https://I /v2/
>> : x509: certificate signed by unknown
>> authority"
>>
>> Is there a way to get that operator to use the same user-ca-bundle?
>>
>
> image import should be using those CAs (it's really about the
> openshift-apiserver, not the samples operator) automatically (sounds like
> another potential bug, but i'll let Oleg weigh in on this one).
>
> However barring that, you can use the mechanism described here to
> setup additional CAs for importing from registries:
>
> https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration
>
> you can follow the more detailed instructions here:
>
> https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca
>

I tried this approach but it didn't work for me.

I ran this command:

oc create configmap registry-cas -n openshift-config \
--from-file=registry.redhat.io..5000=/path/to/ca.crt \
--from-file=registry.redhat.io..443=/path/to/ca.crt \
--from-file=registry.redhat.io=/path/to/ca.crt

and:

oc patch image.config.openshift.io/cluster --patch
'{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge

And that still didn't work. First I deleted the
cluster-samples-operator- pod, then I tried forcing the masters to
restart by touching some machine config (I don't know a better way).
But it still didn't work.  Maybe the samples operator doesn't let you
easily override the trusted CA certs?


>
>
> (Brandi/Adam, we should really include the example from that second link,
> in the general "image resource configuration" page from the first link).
>
> Unfortunately it does not allow you to reuse the user-ca-bundle CM since
> the format of the CM is a bit different (needs an entry per registry
> hostname).
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Mon, Nov 11, 2019 at 11:26 PM Joel Pearson 
wrote:

> I've now discovered that the cluster-samples-operator doesn't seem honour
> the proxy settings, and I see lots of errors in the
> cluster-samples-operator- pod logs
>
> time="2019-11-12T04:15:49Z" level=warning msg="Image import for
> imagestream dotnet tag 2.1 generation 2 failed with detailed message
> Internal error occurred: Get https://registry.redhat.io/v2/: x509:
> certificate signed by unknown authority"
>
> Is there a way to get that operator to use the same user-ca-bundle?
>

image import should be using those CAs (it's really about the
openshift-apiserver, not the samples operator) automatically (sounds like
another potential bug, but i'll let Oleg weigh in on this one).

However barring that, you can use the mechanism described here to
setup additional CAs for importing from registries:
https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-file_image-configuration

you can follow the more detailed instructions here:
https://docs.openshift.com/container-platform/4.2/builds/setting-up-trusted-ca.html#configmap-adding-ca_setting-up-trusted-ca

(Brandi/Adam, we should really include the example from that second link,
in the general "image resource configuration" page from the first link).

Unfortunately it does not allow you to reuse the user-ca-bundle CM since
the format of the CM is a bit different (needs an entry per registry
hostname).



>
> On Tue, 12 Nov 2019 at 14:46, Joel Pearson 
> wrote:
>
>>
>>
>> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>>
>>>
>>>

 Can I use the “trustedCA” part of the proxy configuration without
 actually specifying an explicit proxy?

>>>
>>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>>> consider it a bug).
>>>
>>> It does work! Thanks for that. user-ca-bundle already existed and had my
>> certificate in there, I just needed to reference user-ca-bundle in the
>> proxy config.
>>
>> apiVersion: config.openshift.io/v1
>> kind: Proxy
>> metadata:
>>   name: cluster
>> spec:
>>   trustedCA:
>> name: user-ca-bundle
>>
>
>
>

-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Mon, Nov 11, 2019 at 10:47 PM Joel Pearson 
wrote:

>
>
> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>
>>
>>
>>>
>>> Can I use the “trustedCA” part of the proxy configuration without
>>> actually specifying an explicit proxy?
>>>
>>
>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>> consider it a bug).
>>
>> It does work! Thanks for that. user-ca-bundle already existed and had my
> certificate in there, I just needed to reference user-ca-bundle in the
> proxy config.
>

cool, given that you supplied the CAs during install, and the
user-ca-bundle CM was created, i'm a little surprised the install didn't
automatically setup the reference in the proxyconfig resource for you.  I'm
guessing it did not because there was no actual proxy hostname configured.
I think that's a gap we should close..would you mind filing a bug?  (
bugzilla.redhat.com).  You can submit it against the install component.



>
> apiVersion: config.openshift.io/v1
> kind: Proxy
> metadata:
>   name: cluster
> spec:
>   trustedCA:
> name: user-ca-bundle
>


-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

I've now discovered that the cluster-samples-operator doesn't seem honour
the proxy settings, and I see lots of errors in the
cluster-samples-operator- pod logs

time="2019-11-12T04:15:49Z" level=warning msg="Image import for imagestream
dotnet tag 2.1 generation 2 failed with detailed message Internal error
occurred: Get https://registry.redhat.io/v2/: x509: certificate signed by
unknown authority"

Is there a way to get that operator to use the same user-ca-bundle?

On Tue, 12 Nov 2019 at 14:46, Joel Pearson 
wrote:

>
>
> On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:
>
>>
>>
>>>
>>> Can I use the “trustedCA” part of the proxy configuration without
>>> actually specifying an explicit proxy?
>>>
>>
>> you should be able to.  Daneyon can you confirm?  (if you can't i'd
>> consider it a bug).
>>
>> It does work! Thanks for that. user-ca-bundle already existed and had my
> certificate in there, I just needed to reference user-ca-bundle in the
> proxy config.
>
> apiVersion: config.openshift.io/v1
> kind: Proxy
> metadata:
>   name: cluster
> spec:
>   trustedCA:
> name: user-ca-bundle
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Tue, 12 Nov 2019 at 06:56, Ben Parees  wrote:

>
>
>>
>> Can I use the “trustedCA” part of the proxy configuration without
>> actually specifying an explicit proxy?
>>
>
> you should be able to.  Daneyon can you confirm?  (if you can't i'd
> consider it a bug).
>
> It does work! Thanks for that. user-ca-bundle already existed and had my
certificate in there, I just needed to reference user-ca-bundle in the
proxy config.

apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  name: cluster
spec:
  trustedCA:
name: user-ca-bundle
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Mon, Nov 11, 2019 at 2:51 PM Joel Pearson 
wrote:

>
>
> On Tue, 12 Nov 2019 at 12:26 am, Ben Parees  wrote:
>
>>
>>
>> On Mon, Nov 11, 2019 at 1:17 AM Joel Pearson <
>> japear...@agiledigital.com.au> wrote:
>>
>>> Hi,
>>>
>>> I’m trying to build an image in Openshift 4.2 where my internet has an
>>> MITM proxy.
>>>
>>> So trying to pull docker images fails during the build with x509 errors.
>>>
>>> Is there a way to provide extra trusted CA certificates to the builder?
>>>
>>
>> Did you supply additional CAs via the proxy configuration?  Those should
>> be picked up by the builder automatically when it is pulling images and I
>> think it'd be a bug if you configured that and it's not working:
>>
>> https://docs.openshift.com/container-platform/4.2/networking/enable-cluster-wide-proxy.html#nw-proxy-configure-object_config-cluster-wide-proxy
>>
>
>
>> 
>>
> I forgot to mention that it’s a transparent proxy, in install-config.yaml
> I added the proxy CA to “additionalTrustBundle” which helped it install
> the cluster. But it just didn’t seem to apply to the builder.
>

Hm, i believe it should, Adam can confirm but if it doesn't i'd consider it
a bug.  I know we had a few gaps when 4.2 went out the door, it's possible
this was a known limitation since we provide the first class image config
mechanism to provide additional CAs for builds to use when pulling images.


>
> Can I use the “trustedCA” part of the proxy configuration without
> actually specifying an explicit proxy?
>

you should be able to.  Daneyon can you confirm?  (if you can't i'd
consider it a bug).



> --
> Kind Regards,
>
> Joel Pearson
> Agile Digital | Senior Software Consultant
>
> Love Your Software™ | ABN 98 106 361 273
> p: 1300 858 277 | m: 0405 417 843 <0405417843> | w: agiledigital.com.au
>


-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

On Tue, 12 Nov 2019 at 12:26 am, Ben Parees  wrote:

>
>
> On Mon, Nov 11, 2019 at 1:17 AM Joel Pearson <
> japear...@agiledigital.com.au> wrote:
>
>> Hi,
>>
>> I’m trying to build an image in Openshift 4.2 where my internet has an
>> MITM proxy.
>>
>> So trying to pull docker images fails during the build with x509 errors.
>>
>> Is there a way to provide extra trusted CA certificates to the builder?
>>
>
> Did you supply additional CAs via the proxy configuration?  Those should
> be picked up by the builder automatically when it is pulling images and I
> think it'd be a bug if you configured that and it's not working:
>
> https://docs.openshift.com/container-platform/4.2/networking/enable-cluster-wide-proxy.html#nw-proxy-configure-object_config-cluster-wide-proxy
>


>
I forgot to mention that it’s a transparent proxy, in install-config.yaml I
added the proxy CA to “additionalTrustBundle” which helped it install the
cluster. But it just didn’t seem to apply to the builder.

Can I use the “trustedCA” part of the proxy configuration without actually
specifying an explicit proxy?
-- 
Kind Regards,

Joel Pearson
Agile Digital | Senior Software Consultant

Love Your Software™ | ABN 98 106 361 273
p: 1300 858 277 | m: 0405 417 843 <0405417843> | w: agiledigital.com.au
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use extra trusted CA certs when pulling images for a builder

[Ben Parees]

On Mon, Nov 11, 2019 at 1:17 AM Joel Pearson 
wrote:

> Hi,
>
> I’m trying to build an image in Openshift 4.2 where my internet has an
> MITM proxy.
>
> So trying to pull docker images fails during the build with x509 errors.
>
> Is there a way to provide extra trusted CA certificates to the builder?
>

Did you supply additional CAs via the proxy configuration?  Those should be
picked up by the builder automatically when it is pulling images and I
think it'd be a bug if you configured that and it's not working:
https://docs.openshift.com/container-platform/4.2/networking/enable-cluster-wide-proxy.html#nw-proxy-configure-object_config-cluster-wide-proxy

Barring that, you can also supply additional CAs for trusting registries
(which in the case of your MITM proxy should also be effective) via the
image config resource:
https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-insecure_image-configuration




>
> Pulling image registry.redhat.io/ubi7-minimal:7.7 ...
>
> Warning: Pull failed, retrying in 5s ...
>
> Warning: Pull failed, retrying in 5s ...
>
> Warning: Pull failed, retrying in 5s ...
>
> error: build error: failed to pull image: After retrying 2 times, Pull
> image still failed due to error: while pulling "docker://
> registry.redhat.io/ubi7-minimal:7.7" as "
> registry.redhat.io/ubi7-minimal:7.7": Error initializing source docker://
> registry.redhat.io/ubi7-minimal:7.7: pinging docker registry returned:
> Get https://registry.redhat.io/v2/: x509: certificate signed by unknown
> authority
>
> Thanks,
>
> Joel
>
> --
> Kind Regards,
>
> Joel Pearson
> Agile Digital | Senior Software Consultant
>
> Love Your Software™ | ABN 98 106 361 273
> p: 1300 858 277 | m: 0405 417 843 <0405417843> | w: agiledigital.com.au
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>


-- 
Ben Parees | OpenShift
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

How to use extra trusted CA certs when pulling images for a builder

[Joel Pearson]

Hi,

I’m trying to build an image in Openshift 4.2 where my internet has an MITM
proxy.

So trying to pull docker images fails during the build with x509 errors.

Is there a way to provide extra trusted CA certificates to the builder?

Pulling image registry.redhat.io/ubi7-minimal:7.7 ...

Warning: Pull failed, retrying in 5s ...

Warning: Pull failed, retrying in 5s ...

Warning: Pull failed, retrying in 5s ...

error: build error: failed to pull image: After retrying 2 times, Pull
image still failed due to error: while pulling "docker://
registry.redhat.io/ubi7-minimal:7.7" as "registry.redhat.io/ubi7-minimal:7.7":
Error initializing source docker://registry.redhat.io/ubi7-minimal:7.7:
pinging docker registry returned: Get https://registry.redhat.io/v2/: x509:
certificate signed by unknown authority

Thanks,

Joel

-- 
Kind Regards,

Joel Pearson
Agile Digital | Senior Software Consultant

Love Your Software™ | ABN 98 106 361 273
p: 1300 858 277 | m: 0405 417 843 <0405417843> | w: agiledigital.com.au
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users