Re: [Openstack-operators] [OCTAVIA][QUEENS][KOLLA] - Amphora to Health-manager invalid UDP heartbeat.

[Gaël THEROND]

Hi Michael,

Thanks a lot for those many details regarding the transition between
different states, indeed as you said, my LB passed from pending_update to
active but I still had an offline status this morning as I still received
UDP Packets that HM dropped.

When I was talking about the HealthManager reaching to the amphora on port
9443 of course I didn't mean it use the heartbeat key.


I just had a look at my Amphora and Octavia CP (Control Plan) versions,
seems a little bit off sync as my amphora agent is: *%prog 3.0.0.0b4.dev6*
while my octavia CP services are: *%prog 2.0.1*

I've just updated to stable/rocky this morning and so jumped to: *%prog
3.0.1*
I'll check if I still encounter this issue, but for now my issue seems to
have vanished as I've the following messages:

*2018-10-24 11:58:54.620 24 DEBUG futurist.periodics [-] Submitting
periodic callback 'octavia.cmd.health_manager.periodic_health_check'
_process_scheduled
/usr/lib/python2.7/site-packages/futurist/periodics.py:639*
*2018-10-24 11:58:57.620 24 DEBUG futurist.periodics [-] Submitting
periodic callback 'octavia.cmd.health_manager.periodic_health_check'
_process_scheduled
/usr/lib/python2.7/site-packages/futurist/periodics.py:639*
*2018-10-24 11:59:00.620 24 DEBUG futurist.periodics [-] Submitting
periodic callback 'octavia.cmd.health_manager.periodic_health_check'
_process_scheduled
/usr/lib/python2.7/site-packages/futurist/periodics.py:639*
*2018-10-24 11:59:03.620 24 DEBUG futurist.periodics [-] Submitting
periodic callback 'octavia.cmd.health_manager.periodic_health_check'
_process_scheduled
/usr/lib/python2.7/site-packages/futurist/periodics.py:639*
*2018-10-24 11:59:04.557 23 DEBUG
octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
('172.27.201.105', 48342) dorecv
/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:187*
*2018-10-24 11:59:04.619 45 DEBUG
octavia.controller.healthmanager.health_drivers.update_db [-] Health Update
finished in: 0.0600640773773 seconds update_health
/usr/lib/python2.7/site-packages/octavia/controller/healthmanager/health_drivers/update_db.py:93*

I'll update you with my following investigation, but so far, the issue
seems to be resolve, I'll tweak a bit the timeouts as my LB take a lot
of time to create Listeners/Pools and come to an online status.

Thanks a lot!

Le mar. 23 oct. 2018 à 19:09, Michael Johnson  a
écrit :

> Are the controller and the amphora using the same version of Octavia?
>
> We had a python3 issue where we had to change the HMAC digest used. If
> you controller is running an older version of Octavia than your
> amphora images, it may not have the compatibility code to support the
> new format.  The compatibility code is here:
>
> https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/health_daemon/status_message.py#L56
>
> There is also a release note about the issue here:
> https://docs.openstack.org/releasenotes/octavia/rocky.html#upgrade-notes
>
> If that is not the issue, I would double check the heartbeat_key in
> the health manager configuration files and inside one of the amphora.
>
> Note, that this key is only used for health heartbeats and stats, it
> is not used for the controller to amphora communication on port 9443.
>
> Also, load balancers cannot get "stuck" in PENDING_* states unless
> someone has killed the controller process that was actively working on
> that load balancer. By killed I mean a non-graceful shutdown of the
> process that was in the middle of working on the load balancer.
> Otherwise all code paths lead back to ACTIVE or ERROR status after it
> finishes the work or gives up retrying the requested action. Check
> your controller logs to make sure this load balancer is not still
> being worked on by one of the controllers. The default retry timeouts
> (some are up to 25 minutes) are very long (it will keep trying to
> accomplish the request) to accommodate very slow (virtual box) hosts
> and the test gates. You will want to tune those down for a production
> deployment.
>
> Michael
>
> On Tue, Oct 23, 2018 at 7:09 AM Gaël THEROND 
> wrote:
> >
> > Hi guys,
> >
> > I'm finishing to work on my POC for Octavia and after solving few issues
> with my configuration I'm close to get a properly working setup.
> > However, I'm facing a small but yet annoying bug with the health-manager
> receiving amphora heartbeat UDP packet which it consider as not correct and
> so drop it.
> >
> > Here are the messages that can be found in logs:
> >
> > 2018-10-23 13:53:21.844 25 WARNING
> octavia.amphorae.backends.health_daemon.status_message [-] calculated hmac:
> faf73e41a0f843b826ee581c3995b7f7e56b5e5a294fca0b84eda426766f8415 not equal
> to msg hmac:
> 61376133373164326363653938323764313433373

Re: [Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR

[Gaël THEROND]

For the records, I’m actually working on a fairly large overhaul of our
Openstack services deployement using Kolla-Ansible. We’re leveraging
kolla-ansible to as smoothly as possible migrate all ou legacy architecture
to a shiny new using exactly the same topology as described upper (Using
cumulus/calico etc).

One of the new services that we try to provide with such method is Octavia.

As I too faced some trouble I find them not that hard to solve either by
reading carefully the current APIs ref, guides available and source code or
by asking for help right here.

People responding to octavia’s questions are IMHO blazing fast and really
clear and add great details about internals mechanisms which is really
appreciated.

As I’ve almost finish our own deployment I had noted almost all pitfalls
that I faced and which part of the documentation that was missing.

I’ll finish my deployment and test and redact a clean (and I hope as
complet as possible) documentation as I feel it’s something really needed.

On a side note regarding CA and SSL I had an issue that I solved by
correctly rebuilding my amphora. Another tip and trick here is to use
Barbican when possible as it really help a lot.

I hope it can help anyone else willing to use Octavia as I truly think this
service is a huge addition to Openstack and its gaining more and more
momentum since the Pike/Queens releases.

Le mar. 23 oct. 2018 à 19:49, Michael Johnson  a
écrit :

> I am still catching up on e-mail from the weekend.
>
> There are a lot of different options for how to implement the
> lb-mgmt-network for the controller<->amphora communication. I can't
> talk to what options Kolla provides, but I can talk to how Octavia
> works.
>
> One thing to note on the lb-mgmt-net issue, if you can setup routes
> such that the controllers can reach the IP addresses used for the
> lb-mgmt-net, and that the amphora can reach the controllers, Octavia
> can run with a routed lb-mgmt-net setup. There is no L2 requirement
> between the controllers and the amphora instances.
>
> Michael
>
> On Tue, Oct 23, 2018 at 9:57 AM Erik McCormick
>  wrote:
> >
> > So in your other email you said asked if there was a guide for
> > deploying it with Kolla ansible...
> >
> > Oh boy. No there's not. I don't know if you've seen my recent mails on
> > Octavia, but I am going through this deployment process with
> > kolla-ansible right now and it is lacking in a few areas.
> >
> > If you plan to use different CA certificates for client and server in
> > Octavia, you'll need to add that into the playbook. Presently it only
> > copies over ca_01.pem, cacert.key, and client.pem and uses them for
> > everything. I was completely unable to make it work with only one CA
> > as I got some SSL errors. It passes gate though, so I aasume it must
> > work? I dunno.
> >
> > Networking comments and a really messy kolla-ansible / octavia how-to
> below...
> >
> > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann
> >  wrote:
> > >
> > > Am 10/23/18 um 3:20 PM schrieb Erik McCormick:
> > > > On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann
> > > >  wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> We did test Octavia with Pike (DVR deployment) and everything was
> > > >> working right our of the box. We changed our underlay network to a
> > > >> Layer3 spine-leaf network now and did not deploy DVR as we don't
> wanted
> > > >> to have that much cables in a rack.
> > > >>
> > > >> Octavia is not working right now as the lb-mgmt-net does not exist
> on
> > > >> the compute nodes nor does a br-ex.
> > > >>
> > > >> The control nodes running
> > > >>
> > > >> octavia_worker
> > > >> octavia_housekeeping
> > > >> octavia_health_manager
> > > >> octavia_api
> > > >>
> > > >> and as far as I understood octavia_worker, octavia_housekeeping and
> > > >> octavia_health_manager have to talk to the amphora instances. But
> the
> > > >> control nodes are spread over three different leafs. So each control
> > > >> node in a different L2 domain.
> > > >>
> > > >> So the question is how to deploy a lb-mgmt-net network in our setup?
> > > >>
> > > >> - Compute nodes have no "stretched" L2 domain
> > > >> - Control nodes, compute nodes and network nodes are in L3 networks
> like
> > > >> api, storage, ...
> > > >> - Only network nodes are connected to a L2 domain (with a separated
> NIC)
> > > >> providing the "public" network
> > > >>
> > > > You'll need to add a new bridge to your compute nodes and create a
> > > > provider network associated with that bridge. In my setup this is
> > > > simply a flat network tied to a tagged interface. In your case it
> > > > probably makes more sense to make a new VNI and create a vxlan
> > > > provider network. The routing in your switches should handle the
> rest.
> > >
> > > Ok that's what I try right now. But I don't get how to setup something
> > > like a VxLAN provider Network. I thought only vlan and flat is
> supported
> > > as provider network? I guess it is not possible 

[Openstack-operators] [OCTAVIA][QUEENS][KOLLA] - Amphora to Health-manager invalid UDP heartbeat.

[Gaël THEROND]

Hi guys,

I'm finishing to work on my POC for Octavia and after solving few issues
with my configuration I'm close to get a properly working setup.
However, I'm facing a small but yet annoying bug with the health-manager
receiving amphora heartbeat UDP packet which it consider as not correct and
so drop it.

Here are the messages that can be found in logs:

*2018-10-23 13:53:21.844 25 WARNING
octavia.amphorae.backends.health_daemon.status_message [-] calculated hmac:
faf73e41a0f843b826ee581c3995b7f7e56b5e5a294fca0b84eda426766f8415 not equal
to msg hmac:
6137613337316432636365393832376431343337306537353066626130653261 dropping
packet*

Which come from this part of the HM Code:

https://docs.openstack.org/octavia/pike/_modules/octavia/amphorae/backends/health_daemon/status_message.html#get_payload

The annoying thing is that I don't get why the UDP packet is considered as
stale and how can I try to reproduce the payload which is send to the
HealthManager.
I'm willing to write a simple PY program to simulate the heartbeat payload
but I don't now what's exactly the message and I think I miss some
informations.

Both HealthManager and the Amphora do use the same heartbeat_key and both
can contact on the network as the initial Health-manager to Amphora 9443
connection is validated.

As an effect to this situation, my loadbalancer is stuck in PENDING_UPDATE
mode.

Do you have any idea on how can I handle such thing or if it's something
already seen out there for anyone else?

Kind regards,
G.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface

[Gaël THEROND]

Doing the same documentation process here as well (except that I’m using
kolla). The only annoying thing is the doc submission process :-/.

Le lun. 22 oct. 2018 à 16:50, Erik McCormick  a
écrit :

> Oops, dropped Operators. Can't wait until it's all one list...
> On Mon, Oct 22, 2018 at 10:44 AM Erik McCormick
>  wrote:
> >
> > On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin 
> wrote:
> > >
> > > Hello,
> > >
> > > I've been having a lot of issues with SSL certificates myself, on my
> > > second trip now trying to get it working.
> > >
> > > Before I spent a lot of time walking through every line in the DevStack
> > > plugin and fixing my config options, used the generate
> > > script [1] and still it didn't work.
> > >
> > > When I got the "invalid padding" issue it was because of the DN I used
> > > for the CA and the certificate IIRC.
> > >
> > >  > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
> > > octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
> > > to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
> > > routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
> > > ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
> > > ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> > >  > 19:47 < tobias-urdin> after a quick google "The problem was that my
> > > CA DN was the same as the certificate DN."
> > >
> > > IIRC I think that solved it, but then again I wouldn't remember fully
> > > since I've been at so many different angles by now.
> > >
> > > Here is my IRC logs history from the #openstack-lbaas channel, perhaps
> > > it can help you out
> > > http://paste.openstack.org/show/732575/
> > >
> >
> > Tobias, I owe you a beer. This was precisely the issue. I'm deploying
> > Octavia with kolla-ansible. It only deploys a single CA. After hacking
> > the templates and playbook to incorporate a separate server CA, the
> > amphorae now load and provision the required namespace. I'm adding a
> > kolla tag to the subject of this in hopes that someone might want to
> > take on changing this behavior in the project. Hopefully after I get
> > through Upstream Institute in Berlin I'll be able to do it myself if
> > nobody else wants to do it.
> >
> > For certificate generation, I extracted the contents of
> > octavia_certs_install.yml (which sets up the directory structure,
> > openssl.cnf, and the client CA), and octavia_certs.yml (which creates
> > the server CA and the client certificate) and mashed them into a
> > separate playbook just for this purpose. At the end I get:
> >
> > ca_01.pem - Client CA Certificate
> > ca_01.key - Client CA Key
> > ca_server_01.pem - Server CA Certificate
> > cakey.pem - Server CA Key
> > client.pem - Concatenated Client Key and Certificate
> >
> > If it would help to have the playbook, I can stick it up on github
> > with a huge "This is a hack" disclaimer on it.
> >
> > > -
> > >
> > > Sorry for hijacking the thread but I'm stuck as well.
> > >
> > > I've in the past tried to generate the certificates with [1] but now
> > > moved on to using the openstack-ansible way of generating them [2]
> > > with some modifications.
> > >
> > > Right now I'm just getting: Could not connect to instance. Retrying.:
> > > SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
> > > from the amphoras, haven't got any further but I've eliminated a lot of
> > > stuck in the middle.
> > >
> > > Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
> > > wasn't an issue with CentOS and OpenSSL versions since it tends to lag
> > > behind.
> > > Checking the amphora with openssl s_client [3] it gives the same one,
> > > but the verification is successful just that I don't understand what
> the
> > > bad signature
> > > part is about, from browsing some OpenSSL code it seems to be related
> to
> > > RSA signatures somehow.
> > >
> > > 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
> > > signature:s3_clnt.c:2032:
> > >
> > > So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
> > > (openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
> > > back to something related
> > > to the certificates or the communication between the endpoints, or what
> > > actually responds inside the amphora (gunicorn IIUC?). Based on the
> > > "verify" functions actually causing that bad signature error I would
> > > assume it's the generated certificate that the amphora presents that is
> > > causing it.
> > >
> > > I'll have to continue the troubleshooting to the inside of the amphora,
> > > I've used the test-only amphora image before but have now built my own
> > > one that is
> > > using the amphora-agent from the actual stable branch, but same issue
> > > (bad signature).
> > >
> > > For verbosity this is the config options set for the certificates in
> > > octavia.conf and which file it was copied from [4], same here, a
> > > replication of what 

Re: [Openstack-operators] [Octavia] SSL errors polling amphorae and missing tenant network interface

[Gaël THEROND]

Hi eric!

Glad I’m not the only one having this issue with the ssl communication
between the amphora and the CP.

Even if I don’t yet get a clear answer regarding that issue, I think your
second issue is not an issue as the interface is mounted on a namespace and
so you’ll need to list all nic even those from namespace.

Use an ip netns ls to get the namespace.

Hope it will help.

Le ven. 19 oct. 2018 à 20:40, Erik McCormick  a
écrit :

> I've been wrestling with getting Octavia up and running and have
> become stuck on two issues. I'm hoping someone has run into these
> before. My google foo has come up empty.
>
> Issue 1:
> When the Octavia controller tries to poll the amphora instance, it
> tries repeatedly and eventually fails. The error on the controller
> side is:
>
> 2018-10-19 14:17:39.181 26 ERROR
> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection
> retries (currently set to 300) exhausted.  The amphora is unavailable.
> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries
> exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by
> SSLError(SSLError("bad handshake: Error([('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',
> port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15
> (Caused by SSLError(SSLError("bad handshake: Error([('rsa routines',
> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',
> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding
> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',
> 'tls_process_server_certificate', 'certificate verify
> failed')],)",),))
>
> On the amphora side I see:
> [2018-10-19 17:52:54 +] [1331] [DEBUG] Error processing SSL request.
> [2018-10-19 17:52:54 +] [1331] [DEBUG] Invalid request from
> ip=:::10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake
> failure (_ssl.c:1754)
>
> I've generated certificates both with the script in the Octavia git
> repo, and with the Openstack Ansible playbook. I can see that they are
> present in /etc/octavia/certs.
>
> I'm using the Kolla (Queens) containers for the control plane so I'm
> sure I've satisfied all the python library constraints.
>
> Issue 2:
> I"m not sure how it gets configured, but the tenant network interface
> (ens6) never comes up. I can spawn other instances on that network
> with no issue, and I can see that Neutron has the port attached to the
> instance. However, in the instance this is all I get:
>
> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a
> 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
> group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
>valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
>valid_lft forever preferred_lft forever
> 2: ens3:  mtu 9000 qdisc pfifo_fast
> state UP group default qlen 1000
> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff
> inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3
>valid_lft forever preferred_lft forever
> inet6 fe80::f816:3eff:fe30:c460/64 scope link
>valid_lft forever preferred_lft forever
> 3: ens6:  mtu 1500 qdisc noop state DOWN group
> default qlen 1000
> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff
>
> There's no evidence of the interface anywhere else including udev rules.
>
> Any help with either or both issues would be greatly appreciated.
>
> Cheers,
> Erik
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [OCTAVIA][QUEENS][KOLLA] - network/subnet not found.

[Gaël THEROND]

Hi guys,

I'm back to business with Octavia after a long time but I'm facing an issue
that seems a little bit tricky.

When trying to create a LB using either APIs (cURL/postman) calls or
openstack-client the request finish with an error such as:

`Network c0d40dfd-123e-4a3c-92de-eb7b57178dd3 not found. (HTTP 400)`

If I put my client or the Octavia api in DEBUG mode, I found out neutron to
correctly sending back to him a RESP BODY with the requested network/subnet
in it.

Here is the stacktrace that I get from both, Openstack client and the
Octavia API logs:

```
POST call to
https://api-emea-west-az1.cloud.inkdrop.sh:9876/v2.0/lbaas/loadbalancers
used request id req-2f929192-4e60-491b-b65d-3a7bef43e978
Request returned failure status: 400
Network c0d40dfd-123e-4a3c-92de-eb7b57178dd3 not found. (HTTP 400)
(Request-ID: req-2f929192-4e60-491b-b65d-3a7bef43e978)
Traceback (most recent call last):
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/api/v2/octavia.py",
line 29, in wrapper
response = func(*args, **kwargs)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/api/v2/octavia.py",
line 92, in load_balancer_create
response = self.create(url, **params)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/api/api.py",
line 164, in create
ret = self._request(method, url, session=session, **params)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/api/api.py",
line 141, in _request
return session.request(url, method, **kwargs)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/keystoneauth1/session.py",
line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: Bad Request (HTTP 400)
(Request-ID: req-2f929192-4e60-491b-b65d-3a7bef43e978)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/cliff/app.py",
line 402, in run_subcommand
result = cmd.run(parsed_args)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/command/command.py",
line 41, in run
return super(Command, self).run(parsed_args)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/cliff/display.py",
line 116, in run
column_names, data = self.take_action(parsed_args)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/osc/v2/load_balancer.py",
line 121, in take_action
json=body)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/api/v2/octavia.py",
line 38, in wrapper
request_id=e.request_id)
octaviaclient.api.v2.octavia.OctaviaClientException: Network
c0d40dfd-123e-4a3c-92de-eb7b57178dd3 not found. (HTTP 400) (Request-ID:
req-2f929192-4e60-491b-b65d-3a7bef43e978)
clean_up CreateLoadBalancer: Network c0d40dfd-123e-4a3c-92de-eb7b57178dd3
not found. (HTTP 400) (Request-ID: req-2f929192-4e60-491b-b65d-3a7bef43e978)
Traceback (most recent call last):
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/api/v2/octavia.py",
line 29, in wrapper
response = func(*args, **kwargs)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/octaviaclient/api/v2/octavia.py",
line 92, in load_balancer_create
response = self.create(url, **params)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/api/api.py",
line 164, in create
ret = self._request(method, url, session=session, **params)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/api/api.py",
line 141, in _request
return session.request(url, method, **kwargs)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/keystoneauth1/session.py",
line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: Bad Request (HTTP 400)
(Request-ID: req-2f929192-4e60-491b-b65d-3a7bef43e978)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/shell.py",
line 135, in run
ret_val = super(OpenStackShell, self).run(argv)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/cliff/app.py",
line 281, in run
result = self.run_subcommand(remainder)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/shell.py",
line 175, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/cliff/app.py",
line 402, in run_subcommand
result = cmd.run(parsed_args)
  File
"/home/flint/.virtualenvs/ic-emea-az1/lib/python3.4/site-packages/osc_lib/command/command.py",
line 41, in