Hi Flemming,
certmgr at the moment doesn't support fullchain PEM certificates:
https://tracker.ceph.com/issues/75710
Can you plz paste the spec you are currently using that is leading to the
issue). Right now the format of cert/key has to be something like:
service_type: rgw
service_id: myrgw
spec:
ssl: true
ssl_key: |
-----BEGIN PRIVATE KEY-----
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCM58IqqXGTCJof
.....
QsJ69XOQV9Os37zG2/hELQ1ycBs2Jw==
-----END PRIVATE KEY-----
ssl_cert: |
-----BEGIN CERTIFICATE-----
MIIF4TCCA8mgAwIBAgIUI1F5+FeKTeF5J+xYLdkEDP9J0OMwDQYJKoZIhvcNAQEL
....
fhTw47C8Dry+2bOaYQPe6qvoFQ+x
-----END CERTIFICATE-----
Best,
Redo.
On Fri, Apr 24, 2026 at 2:52 PM Flemming Frandsen via ceph-users <
[email protected]> wrote:
> I have a tentacle 20.2.1 cluster managed by cephadm, where I have
> configured a certificate for the rgw using something like:
>
> service_type: ingress
> service_id: rgw.default
> service_name: ingress.rgw.default
> placement:
> count: 3
> label: rgw
> spec:
> backend_service: rgw.default
> first_virtual_router_id: 50
> frontend_port: 443
> monitor_port: 8080
> virtual_ip: 10.0.40.3/24
> networks:
> - 10.0.40.0/24
> ssl_cert: |
> -----BEGIN CERTIFICATE-----
> *snip*
> -----END CERTIFICATE-----
> -----BEGIN CERTIFICATE-----
> *snip*
> -----END CERTIFICATE-----
> -----BEGIN RSA PRIVATE KEY-----
> *snip*
> -----END RSA PRIVATE KEY-----
>
> This initially worked perfectly, but I have just rotated the certificate by
> generating the yaml with a fresh private key and associated certificate
> then installing it with:
> ceph orch apply -i new.yaml
>
> Clients accessing the rgw ingress get the correct certificate and work
> perfectly, both before and after the key rotation.
>
>
> I have a script that does this for me daily, and until the certificate was
> rotated, everything was fine, but then I got this error:
>
> # ceph orch certmgr cert check
>
> Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not
> valid (error: Invalid cert/key pair: [('x509 certificate routines', '',
> 'key values mismatch')])
>
> After messing with the certificate a bunch I now get a completely
> different, but also worrying error:
>
> # ceph orch certmgr cert check
> - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not
> valid (error: Invalid private key: [('DECODER routines', '',
> 'unsupported')])
>
> The documentation offers no help:
> https://docs.ceph.com/en/tentacle/cephadm/certmgr/
>
> The error seems to come from this code:
>
> https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/internal.py#L134
>
> So the error is an amputated exception from somewhere in openssl but the
> actual exception is eaten by the code, so there is no way for me to debug
> the situation:
>
> https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/internal.py#L134
>
> I have tried installing the old certificate and key, that only leads the
> the Invalid private key: [('DECODER routines', '', 'unsupported')] error.
>
> I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does
> nothing.
>
> I have even tried copying the key and cert pem files into the mgr container
> and verified them using the openssl installed in the container.
>
> The certificates are generated by cert-manager, running in k8s and a
> cronjob that periodically pushes the new key/cert to the ceph cluster via
> ssh.
>
> Has anyone seen these types of errors before?
>
> --
> Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
