Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
FYI, I took some time out this afternoon and wrote a detailed
certificate configuration guide. Hopefully this will help.
https://review.openstack.org/613454
Reviews would be welcome!
Michael
On Thu, Oct 25, 2018 at 7:00 AM Tobias Urdin wrote:
>
> Might as well throw it out here.
>
> After a lot of troubleshooting we were able to narrow our issue down to
> our test environment running qemu virtualization, we moved our compute
> node to hardware and
> used kvm full virtualization instead.
>
> We could properly reproduce the issue where generating a CSR from a
> private key and then trying to verify the CSR would fail complaining about
> "Signature did not match the certificate request"
>
> We suspect qemu floating point emulation caused this, the same OpenSSL
> function that validates a CSR is the one used when validating the SSL
> handshake which caused our issue.
> After going through the whole stack, we have Octavia working flawlessly
> without any issues at all.
>
> Best regards
> Tobias
>
> On 10/23/2018 04:31 PM, Tobias Urdin wrote:
> > Hello Erik,
> >
> > Could you specify the DNs you used for all certificates just so that I
> > can rule it out on my side.
> > You can redact anything sensitive with some to just get the feel on how
> > it's configured.
> >
> > Best regards
> > Tobias
> >
> > On 10/22/2018 04:47 PM, Erik McCormick wrote:
> >> On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin
> >> wrote:
> >>> Hello,
> >>>
> >>> I've been having a lot of issues with SSL certificates myself, on my
> >>> second trip now trying to get it working.
> >>>
> >>> Before I spent a lot of time walking through every line in the DevStack
> >>> plugin and fixing my config options, used the generate
> >>> script [1] and still it didn't work.
> >>>
> >>> When I got the "invalid padding" issue it was because of the DN I used
> >>> for the CA and the certificate IIRC.
> >>>
> >>>> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
> >>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
> >>> to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
> >>> routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
> >>> ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
> >>> ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> >>>> 19:47 < tobias-urdin> after a quick google "The problem was that my
> >>> CA DN was the same as the certificate DN."
> >>>
> >>> IIRC I think that solved it, but then again I wouldn't remember fully
> >>> since I've been at so many different angles by now.
> >>>
> >>> Here is my IRC logs history from the #openstack-lbaas channel, perhaps
> >>> it can help you out
> >>> http://paste.openstack.org/show/732575/
> >>>
> >> Tobias, I owe you a beer. This was precisely the issue. I'm deploying
> >> Octavia with kolla-ansible. It only deploys a single CA. After hacking
> >> the templates and playbook to incorporate a separate server CA, the
> >> amphorae now load and provision the required namespace. I'm adding a
> >> kolla tag to the subject of this in hopes that someone might want to
> >> take on changing this behavior in the project. Hopefully after I get
> >> through Upstream Institute in Berlin I'll be able to do it myself if
> >> nobody else wants to do it.
> >>
> >> For certificate generation, I extracted the contents of
> >> octavia_certs_install.yml (which sets up the directory structure,
> >> openssl.cnf, and the client CA), and octavia_certs.yml (which creates
> >> the server CA and the client certificate) and mashed them into a
> >> separate playbook just for this purpose. At the end I get:
> >>
> >> ca_01.pem - Client CA Certificate
> >> ca_01.key - Client CA Key
> >> ca_server_01.pem - Server CA Certificate
> >> cakey.pem - Server CA Key
> >> client.pem - Concatenated Client Key and Certificate
> >>
> >> If it would help to have the playbook, I can stick it up on github
> >> with a huge "This is a hack" disclaimer on it.
> >>
> >>> -
> >>>
> >>> Sorry for hijacking the thread but I'm stuck as well.
> >>>
> >>> I've in the past tried to generate the certificates with [1] but now
> >>> moved on to using the openstack-ansible way of generating them [2]
> >>> with some modifications.
> >>>
> >>> Right now I'm just getting: Could not connect to instance. Retrying.:
> >>> SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
> >>> from the amphoras, haven't got any further but I've eliminated a lot of
> >>> stuck in the middle.
> >>>
> >>> Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
> >>> wasn't an issue with CentOS and OpenSSL versions since it tends to lag
> >>> behind.
> >>> Checking the amphora with openssl s_client [3] it gives the same one,
> >>> but the verification is successful just that I don't understand what the
> >>> bad signature
> >>> part is about, from browsing some OpenSSL code it seems to be related to
> >>> RSA signatures somehow.
> >>>
> >>>
Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
Might as well throw it out here.
After a lot of troubleshooting we were able to narrow our issue down to
our test environment running qemu virtualization, we moved our compute
node to hardware and
used kvm full virtualization instead.
We could properly reproduce the issue where generating a CSR from a
private key and then trying to verify the CSR would fail complaining about
"Signature did not match the certificate request"
We suspect qemu floating point emulation caused this, the same OpenSSL
function that validates a CSR is the one used when validating the SSL
handshake which caused our issue.
After going through the whole stack, we have Octavia working flawlessly
without any issues at all.
Best regards
Tobias
On 10/23/2018 04:31 PM, Tobias Urdin wrote:
Hello Erik,
Could you specify the DNs you used for all certificates just so that I
can rule it out on my side.
You can redact anything sensitive with some to just get the feel on how
it's configured.
Best regards
Tobias
On 10/22/2018 04:47 PM, Erik McCormick wrote:
On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin wrote:
Hello,
I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.
Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.
When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.
> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."
IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.
Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/
Tobias, I owe you a beer. This was precisely the issue. I'm deploying
Octavia with kolla-ansible. It only deploys a single CA. After hacking
the templates and playbook to incorporate a separate server CA, the
amphorae now load and provision the required namespace. I'm adding a
kolla tag to the subject of this in hopes that someone might want to
take on changing this behavior in the project. Hopefully after I get
through Upstream Institute in Berlin I'll be able to do it myself if
nobody else wants to do it.
For certificate generation, I extracted the contents of
octavia_certs_install.yml (which sets up the directory structure,
openssl.cnf, and the client CA), and octavia_certs.yml (which creates
the server CA and the client certificate) and mashed them into a
separate playbook just for this purpose. At the end I get:
ca_01.pem - Client CA Certificate
ca_01.key - Client CA Key
ca_server_01.pem - Server CA Certificate
cakey.pem - Server CA Key
client.pem - Concatenated Client Key and Certificate
If it would help to have the playbook, I can stick it up on github
with a huge "This is a hack" disclaimer on it.
-
Sorry for hijacking the thread but I'm stuck as well.
I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.
Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.
Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.
140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:
So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.
I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual
Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
On Tue, Oct 23, 2018 at 10:20 AM Tobias Urdin wrote:
>
> Hello Erik,
>
> Could you specify the DNs you used for all certificates just so that I
> can rule it out on my side.
> You can redact anything sensitive with some to just get the feel on how
> it's configured.
>
> Best regards
> Tobias
>
I'm not actually using anything special or custom. For right now I
just let it use the default www.example.com stuff. These are the
settings in the playbook which I distilled from OSA
octavia_cert_key_length_server: '4096' # key length
octavia_cert_cipher_server: 'aes256'
octavia_cert_cipher_client: 'aes256'
octavia_cert_key_length_client: '4096' # key length
octavia_cert_server_ca_subject:
'/C=US/ST=Denial/L=Nowhere/O=Dis/CN=www.example.com' # change this to
something more real
octavia_cert_client_ca_subject:
'/C=US/ST=Denial/L=Nowhere/O=Dis/CN=www.example.com' # change this to
something more real
octavia_cert_client_req_common_name: 'www.example.com' # change this
to something more real
octavia_cert_client_req_country_name: 'US'
octavia_cert_client_req_state_or_province_name: 'Denial'
octavia_cert_client_req_locality_name: 'Nowhere'
octavia_cert_client_req_organization_name: 'Dis'
octavia_cert_validity_days: 1825 # 5 years
-Erik
> On 10/22/2018 04:47 PM, Erik McCormick wrote:
> > On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin wrote:
> >> Hello,
> >>
> >> I've been having a lot of issues with SSL certificates myself, on my
> >> second trip now trying to get it working.
> >>
> >> Before I spent a lot of time walking through every line in the DevStack
> >> plugin and fixing my config options, used the generate
> >> script [1] and still it didn't work.
> >>
> >> When I got the "invalid padding" issue it was because of the DN I used
> >> for the CA and the certificate IIRC.
> >>
> >> > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
> >> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
> >> to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
> >> routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
> >> ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
> >> ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> >> > 19:47 < tobias-urdin> after a quick google "The problem was that my
> >> CA DN was the same as the certificate DN."
> >>
> >> IIRC I think that solved it, but then again I wouldn't remember fully
> >> since I've been at so many different angles by now.
> >>
> >> Here is my IRC logs history from the #openstack-lbaas channel, perhaps
> >> it can help you out
> >> http://paste.openstack.org/show/732575/
> >>
> > Tobias, I owe you a beer. This was precisely the issue. I'm deploying
> > Octavia with kolla-ansible. It only deploys a single CA. After hacking
> > the templates and playbook to incorporate a separate server CA, the
> > amphorae now load and provision the required namespace. I'm adding a
> > kolla tag to the subject of this in hopes that someone might want to
> > take on changing this behavior in the project. Hopefully after I get
> > through Upstream Institute in Berlin I'll be able to do it myself if
> > nobody else wants to do it.
> >
> > For certificate generation, I extracted the contents of
> > octavia_certs_install.yml (which sets up the directory structure,
> > openssl.cnf, and the client CA), and octavia_certs.yml (which creates
> > the server CA and the client certificate) and mashed them into a
> > separate playbook just for this purpose. At the end I get:
> >
> > ca_01.pem - Client CA Certificate
> > ca_01.key - Client CA Key
> > ca_server_01.pem - Server CA Certificate
> > cakey.pem - Server CA Key
> > client.pem - Concatenated Client Key and Certificate
> >
> > If it would help to have the playbook, I can stick it up on github
> > with a huge "This is a hack" disclaimer on it.
> >
> >> -
> >>
> >> Sorry for hijacking the thread but I'm stuck as well.
> >>
> >> I've in the past tried to generate the certificates with [1] but now
> >> moved on to using the openstack-ansible way of generating them [2]
> >> with some modifications.
> >>
> >> Right now I'm just getting: Could not connect to instance. Retrying.:
> >> SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
> >> from the amphoras, haven't got any further but I've eliminated a lot of
> >> stuck in the middle.
> >>
> >> Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
> >> wasn't an issue with CentOS and OpenSSL versions since it tends to lag
> >> behind.
> >> Checking the amphora with openssl s_client [3] it gives the same one,
> >> but the verification is successful just that I don't understand what the
> >> bad signature
> >> part is about, from browsing some OpenSSL code it seems to be related to
> >> RSA signatures somehow.
> >>
> >> 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
> >> signature:s3_clnt.c:2032:
> >>
> >> So I've basicly ruled out Ubuntu (openssl-1.1.0g) and
Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
Hello Erik,
Could you specify the DNs you used for all certificates just so that I
can rule it out on my side.
You can redact anything sensitive with some to just get the feel on how
it's configured.
Best regards
Tobias
On 10/22/2018 04:47 PM, Erik McCormick wrote:
On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin wrote:
Hello,
I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.
Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.
When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.
> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."
IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.
Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/
Tobias, I owe you a beer. This was precisely the issue. I'm deploying
Octavia with kolla-ansible. It only deploys a single CA. After hacking
the templates and playbook to incorporate a separate server CA, the
amphorae now load and provision the required namespace. I'm adding a
kolla tag to the subject of this in hopes that someone might want to
take on changing this behavior in the project. Hopefully after I get
through Upstream Institute in Berlin I'll be able to do it myself if
nobody else wants to do it.
For certificate generation, I extracted the contents of
octavia_certs_install.yml (which sets up the directory structure,
openssl.cnf, and the client CA), and octavia_certs.yml (which creates
the server CA and the client certificate) and mashed them into a
separate playbook just for this purpose. At the end I get:
ca_01.pem - Client CA Certificate
ca_01.key - Client CA Key
ca_server_01.pem - Server CA Certificate
cakey.pem - Server CA Key
client.pem - Concatenated Client Key and Certificate
If it would help to have the playbook, I can stick it up on github
with a huge "This is a hack" disclaimer on it.
-
Sorry for hijacking the thread but I'm stuck as well.
I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.
Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.
Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.
140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:
So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.
I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual stable branch, but same issue
(bad signature).
For verbosity this is the config options set for the certificates in
octavia.conf and which file it was copied from [4], same here, a
replication of what openstack-ansible does.
Appreciate any feedback or help :)
Best regards
Tobias
[1]
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/
On 10/20/2018 01:53 AM, Michael Johnson wrote:
Hi Erik,
Sorry to hear you are still having certificate issues.
Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is
Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
Oops, dropped Operators. Can't wait until it's all one list...
On Mon, Oct 22, 2018 at 10:44 AM Erik McCormick
wrote:
>
> On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin wrote:
> >
> > Hello,
> >
> > I've been having a lot of issues with SSL certificates myself, on my
> > second trip now trying to get it working.
> >
> > Before I spent a lot of time walking through every line in the DevStack
> > plugin and fixing my config options, used the generate
> > script [1] and still it didn't work.
> >
> > When I got the "invalid padding" issue it was because of the DN I used
> > for the CA and the certificate IIRC.
> >
> > > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
> > octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
> > to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
> > routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
> > ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
> > ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> > > 19:47 < tobias-urdin> after a quick google "The problem was that my
> > CA DN was the same as the certificate DN."
> >
> > IIRC I think that solved it, but then again I wouldn't remember fully
> > since I've been at so many different angles by now.
> >
> > Here is my IRC logs history from the #openstack-lbaas channel, perhaps
> > it can help you out
> > http://paste.openstack.org/show/732575/
> >
>
> Tobias, I owe you a beer. This was precisely the issue. I'm deploying
> Octavia with kolla-ansible. It only deploys a single CA. After hacking
> the templates and playbook to incorporate a separate server CA, the
> amphorae now load and provision the required namespace. I'm adding a
> kolla tag to the subject of this in hopes that someone might want to
> take on changing this behavior in the project. Hopefully after I get
> through Upstream Institute in Berlin I'll be able to do it myself if
> nobody else wants to do it.
>
> For certificate generation, I extracted the contents of
> octavia_certs_install.yml (which sets up the directory structure,
> openssl.cnf, and the client CA), and octavia_certs.yml (which creates
> the server CA and the client certificate) and mashed them into a
> separate playbook just for this purpose. At the end I get:
>
> ca_01.pem - Client CA Certificate
> ca_01.key - Client CA Key
> ca_server_01.pem - Server CA Certificate
> cakey.pem - Server CA Key
> client.pem - Concatenated Client Key and Certificate
>
> If it would help to have the playbook, I can stick it up on github
> with a huge "This is a hack" disclaimer on it.
>
> > -
> >
> > Sorry for hijacking the thread but I'm stuck as well.
> >
> > I've in the past tried to generate the certificates with [1] but now
> > moved on to using the openstack-ansible way of generating them [2]
> > with some modifications.
> >
> > Right now I'm just getting: Could not connect to instance. Retrying.:
> > SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
> > from the amphoras, haven't got any further but I've eliminated a lot of
> > stuck in the middle.
> >
> > Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
> > wasn't an issue with CentOS and OpenSSL versions since it tends to lag
> > behind.
> > Checking the amphora with openssl s_client [3] it gives the same one,
> > but the verification is successful just that I don't understand what the
> > bad signature
> > part is about, from browsing some OpenSSL code it seems to be related to
> > RSA signatures somehow.
> >
> > 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
> > signature:s3_clnt.c:2032:
> >
> > So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
> > (openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
> > back to something related
> > to the certificates or the communication between the endpoints, or what
> > actually responds inside the amphora (gunicorn IIUC?). Based on the
> > "verify" functions actually causing that bad signature error I would
> > assume it's the generated certificate that the amphora presents that is
> > causing it.
> >
> > I'll have to continue the troubleshooting to the inside of the amphora,
> > I've used the test-only amphora image before but have now built my own
> > one that is
> > using the amphora-agent from the actual stable branch, but same issue
> > (bad signature).
> >
> > For verbosity this is the config options set for the certificates in
> > octavia.conf and which file it was copied from [4], same here, a
> > replication of what openstack-ansible does.
> >
> > Appreciate any feedback or help :)
> >
> > Best regards
> > Tobias
> >
> > [1]
> > https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
> > [2] http://paste.openstack.org/show/732483/
> > [3] http://paste.openstack.org/show/732486/
> > [4] http://paste.openstack.org/show/732487/
> >
> > On 10/20/2018 01:53 AM, Michael Johnson wrote:
> > > Hi Erik,
> > >
> >
Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface
On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin wrote:
>
> Hello,
>
> I've been having a lot of issues with SSL certificates myself, on my
> second trip now trying to get it working.
>
> Before I spent a lot of time walking through every line in the DevStack
> plugin and fixing my config options, used the generate
> script [1] and still it didn't work.
>
> When I got the "invalid padding" issue it was because of the DN I used
> for the CA and the certificate IIRC.
>
> > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
> to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
> routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
> ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
> ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> > 19:47 < tobias-urdin> after a quick google "The problem was that my
> CA DN was the same as the certificate DN."
>
> IIRC I think that solved it, but then again I wouldn't remember fully
> since I've been at so many different angles by now.
>
> Here is my IRC logs history from the #openstack-lbaas channel, perhaps
> it can help you out
> http://paste.openstack.org/show/732575/
>
Tobias, I owe you a beer. This was precisely the issue. I'm deploying
Octavia with kolla-ansible. It only deploys a single CA. After hacking
the templates and playbook to incorporate a separate server CA, the
amphorae now load and provision the required namespace. I'm adding a
kolla tag to the subject of this in hopes that someone might want to
take on changing this behavior in the project. Hopefully after I get
through Upstream Institute in Berlin I'll be able to do it myself if
nobody else wants to do it.
For certificate generation, I extracted the contents of
octavia_certs_install.yml (which sets up the directory structure,
openssl.cnf, and the client CA), and octavia_certs.yml (which creates
the server CA and the client certificate) and mashed them into a
separate playbook just for this purpose. At the end I get:
ca_01.pem - Client CA Certificate
ca_01.key - Client CA Key
ca_server_01.pem - Server CA Certificate
cakey.pem - Server CA Key
client.pem - Concatenated Client Key and Certificate
If it would help to have the playbook, I can stick it up on github
with a huge "This is a hack" disclaimer on it.
> -
>
> Sorry for hijacking the thread but I'm stuck as well.
>
> I've in the past tried to generate the certificates with [1] but now
> moved on to using the openstack-ansible way of generating them [2]
> with some modifications.
>
> Right now I'm just getting: Could not connect to instance. Retrying.:
> SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
> from the amphoras, haven't got any further but I've eliminated a lot of
> stuck in the middle.
>
> Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
> wasn't an issue with CentOS and OpenSSL versions since it tends to lag
> behind.
> Checking the amphora with openssl s_client [3] it gives the same one,
> but the verification is successful just that I don't understand what the
> bad signature
> part is about, from browsing some OpenSSL code it seems to be related to
> RSA signatures somehow.
>
> 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
> signature:s3_clnt.c:2032:
>
> So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
> (openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
> back to something related
> to the certificates or the communication between the endpoints, or what
> actually responds inside the amphora (gunicorn IIUC?). Based on the
> "verify" functions actually causing that bad signature error I would
> assume it's the generated certificate that the amphora presents that is
> causing it.
>
> I'll have to continue the troubleshooting to the inside of the amphora,
> I've used the test-only amphora image before but have now built my own
> one that is
> using the amphora-agent from the actual stable branch, but same issue
> (bad signature).
>
> For verbosity this is the config options set for the certificates in
> octavia.conf and which file it was copied from [4], same here, a
> replication of what openstack-ansible does.
>
> Appreciate any feedback or help :)
>
> Best regards
> Tobias
>
> [1]
> https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
> [2] http://paste.openstack.org/show/732483/
> [3] http://paste.openstack.org/show/732486/
> [4] http://paste.openstack.org/show/732487/
>
> On 10/20/2018 01:53 AM, Michael Johnson wrote:
> > Hi Erik,
> >
> > Sorry to hear you are still having certificate issues.
> >
> > Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
> > network for the VIP, one of the first steps after the worker connects
> > to the amphora agent is finishing the required configuration of the
> > VIP interface inside the network namespace on the amphroa.
