Re: [openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface
+operators
My bad.
On 10/22/2018 10:22 AM, Tobias Urdin wrote:
Hello,
I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.
Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.
When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.
> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."
IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.
Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/
-
Sorry for hijacking the thread but I'm stuck as well.
I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.
Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.
Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.
140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:
So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.
I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual stable branch, but same issue
(bad signature).
For verbosity this is the config options set for the certificates in
octavia.conf and which file it was copied from [4], same here, a
replication of what openstack-ansible does.
Appreciate any feedback or help :)
Best regards
Tobias
[1]
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/
On 10/20/2018 01:53 AM, Michael Johnson wrote:
Hi Erik,
Sorry to hear you are still having certificate issues.
Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is finishing the required configuration of the
VIP interface inside the network namespace on the amphroa.
If I remember correctly, you are attempting to configure Octavia with
the dual CA option (which is good for non-development use).
This is what I have for notes:
[certificates] gets the following:
cert_generator = local_cert_generator
ca_certificate = server CA's "server.pem" file
ca_private_key = server CA's "server.key" file
ca_private_key_passphrase = pass phrase for ca_private_key
[controller_worker]
client_ca = Client CA's ca_cert file
[haproxy_amphora]
client_cert = Client CA's client.pem file (I think with it's key
concatenated is what rm_work said the other day)
server_ca = Server CA's ca_cert file
That said, I can probably run through this and write something up next
week that is more step-by-step/detailed.
Michael
On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick
wrote:
Apologies for cross-posting, but in the event that these might be
worth filing as bugs, I wanted the Octavia devs to see it as well...
I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.
Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and eventually fails. The error on the controller
side is:
2018-10-19 14:17:39.181 26
Re: [openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface
Hello,
I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.
Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.
When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.
> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."
IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.
Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/
-
Sorry for hijacking the thread but I'm stuck as well.
I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.
Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.
Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.
140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:
So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.
I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual stable branch, but same issue
(bad signature).
For verbosity this is the config options set for the certificates in
octavia.conf and which file it was copied from [4], same here, a
replication of what openstack-ansible does.
Appreciate any feedback or help :)
Best regards
Tobias
[1]
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/
On 10/20/2018 01:53 AM, Michael Johnson wrote:
Hi Erik,
Sorry to hear you are still having certificate issues.
Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is finishing the required configuration of the
VIP interface inside the network namespace on the amphroa.
If I remember correctly, you are attempting to configure Octavia with
the dual CA option (which is good for non-development use).
This is what I have for notes:
[certificates] gets the following:
cert_generator = local_cert_generator
ca_certificate = server CA's "server.pem" file
ca_private_key = server CA's "server.key" file
ca_private_key_passphrase = pass phrase for ca_private_key
[controller_worker]
client_ca = Client CA's ca_cert file
[haproxy_amphora]
client_cert = Client CA's client.pem file (I think with it's key
concatenated is what rm_work said the other day)
server_ca = Server CA's ca_cert file
That said, I can probably run through this and write something up next
week that is more step-by-step/detailed.
Michael
On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick
wrote:
Apologies for cross-posting, but in the event that these might be
worth filing as bugs, I wanted the Octavia devs to see it as well...
I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.
Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and eventually fails. The error on the controller
side is:
2018-10-19 14:17:39.181 26 ERROR
Re: [openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface
Hi Erik,
Sorry to hear you are still having certificate issues.
Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is finishing the required configuration of the
VIP interface inside the network namespace on the amphroa.
If I remember correctly, you are attempting to configure Octavia with
the dual CA option (which is good for non-development use).
This is what I have for notes:
[certificates] gets the following:
cert_generator = local_cert_generator
ca_certificate = server CA's "server.pem" file
ca_private_key = server CA's "server.key" file
ca_private_key_passphrase = pass phrase for ca_private_key
[controller_worker]
client_ca = Client CA's ca_cert file
[haproxy_amphora]
client_cert = Client CA's client.pem file (I think with it's key
concatenated is what rm_work said the other day)
server_ca = Server CA's ca_cert file
That said, I can probably run through this and write something up next
week that is more step-by-step/detailed.
Michael
On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick
wrote:
>
> Apologies for cross-posting, but in the event that these might be
> worth filing as bugs, I wanted the Octavia devs to see it as well...
>
> I've been wrestling with getting Octavia up and running and have
> become stuck on two issues. I'm hoping someone has run into these
> before. My google foo has come up empty.
>
> Issue 1:
> When the Octavia controller tries to poll the amphora instance, it
> tries repeatedly and eventually fails. The error on the controller
> side is:
>
> 2018-10-19 14:17:39.181 26 ERROR
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
> retries (currently set to 300) exhausted. The amphora is unavailable.
> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
> exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by
> SSLError(SSLError("bad handshake: Error([('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
> port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15
> (Caused by SSLError(SSLError("bad handshake: Error([('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),))
>
> On the amphora side I see:
> [2018-10-19 17:52:54 +] [1331] [DEBUG] Error processing SSL request.
> [2018-10-19 17:52:54 +] [1331] [DEBUG] Invalid request from
> ip=:::10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake
> failure (_ssl.c:1754)
>
> I've generated certificates both with the script in the Octavia git
> repo, and with the Openstack Ansible playbook. I can see that they are
> present in /etc/octavia/certs.
>
> I'm using the Kolla (Queens) containers for the control plane so I'm
> sure I've satisfied all the python library constraints.
>
> Issue 2:
> I"m not sure how it gets configured, but the tenant network interface
> (ens6) never comes up. I can spawn other instances on that network
> with no issue, and I can see that Neutron has the port attached to the
> instance. However, in the instance this is all I get:
>
> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
> 1: lo: mtu 65536 qdisc noqueue state UNKNOWN
> group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
>valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
>valid_lft forever preferred_lft forever
> 2: ens3: mtu 9000 qdisc pfifo_fast
> state UP group default qlen 1000
> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
> inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3
>valid_lft forever preferred_lft forever
> inet6 fe80::f816:3eff:fe30:c460/64 scope link
>valid_lft forever preferred_lft forever
> 3: ens6: mtu 1500 qdisc noop state DOWN group
> default qlen 1000
> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff
>
> There's no evidence of the interface anywhere else including udev rules.
>
> Any help with either or both issues would be greatly appreciated.
>
> Cheers,
> Erik
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage
[openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface
Apologies for cross-posting, but in the event that these might be
worth filing as bugs, I wanted the Octavia devs to see it as well...
I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.
Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and eventually fails. The error on the controller
side is:
2018-10-19 14:17:39.181 26 ERROR
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
retries (currently set to 300) exhausted. The amphora is unavailable.
Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by
SSLError(SSLError("bad handshake: Error([('rsa routines',
'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
'tls_process_server_certificate', 'certificate verify
failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15
(Caused by SSLError(SSLError("bad handshake: Error([('rsa routines',
'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
'tls_process_server_certificate', 'certificate verify
failed')],)",),))
On the amphora side I see:
[2018-10-19 17:52:54 +] [1331] [DEBUG] Error processing SSL request.
[2018-10-19 17:52:54 +] [1331] [DEBUG] Invalid request from
ip=:::10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake
failure (_ssl.c:1754)
I've generated certificates both with the script in the Octavia git
repo, and with the Openstack Ansible playbook. I can see that they are
present in /etc/octavia/certs.
I'm using the Kolla (Queens) containers for the control plane so I'm
sure I've satisfied all the python library constraints.
Issue 2:
I"m not sure how it gets configured, but the tenant network interface
(ens6) never comes up. I can spawn other instances on that network
with no issue, and I can see that Neutron has the port attached to the
instance. However, in the instance this is all I get:
ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: mtu 9000 qdisc pfifo_fast
state UP group default qlen 1000
link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe30:c460/64 scope link
valid_lft forever preferred_lft forever
3: ens6: mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff
There's no evidence of the interface anywhere else including udev rules.
Any help with either or both issues would be greatly appreciated.
Cheers,
Erik
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
