Stephen visited us today (the joy of spending some days in Seattle☺) and we 
discussed that  further (and sorry for using VM – not sure what won):

1.       We will only support one driver per controller, e.g. if you upgrade a 
driver you deploy a new controller with the new driver and either make him take 
over existing VMs (minor change) or spin  up new ones (major change) but keep 
the “old” controller in place until it doesn’t serve any VMs any longer

2.       If we render configuration files on the VM we only support one upgrade 
model (replacing the VM) which might simplify development as opposed to the 
driver model where we need to write code to push out configuration changes to 
all VMs for minor changes + write code to failover VMs for major changes

3.       I am afraid that half baked drivers will break the controller and I 
feel it’s easier to shoot VMs with half baked renderers  than the controllers.

4.       The main advantage by using an Octavia format to talk to VMs is that 
we can mix and match VMs with different properties (e.g. nginx, haproxy) on the 
same controller because the implementation detail (which file to render) is 

5.       The major difference in The API between Stephen and me would be that I 
would send json files which get rendered on the VM into a haproxy file whereas 
he would send an haproxy file. We still need to develop an interface on the VM 
to report stats and health in Octavia format. It is conceivable with Stephen’s 
design that drivers would exist which would translate stats and health from a 
proprietary format into the Octavia one. I am not sure how we would get the 
proprietary VMs to emit the UDP health packets… In any case a lot of logic 
could end up in a driver – and fanning that processing out to the VMs might 
allow for less controllers.

Overall, if I don’t like to take advantage of the minor update model the main 
difference between me and Stephen is in the haproxy case to ship json instead 
of haproxy config. I understand that the minor update capability is a make or 
break for Stephen though in my experience configuration changes without other 
updates are rare (and my experience might not be representative).

In any case there certainly is some advantage for appliances which not 
necessarily speak the Octavia protocol by allowing custom drivers. However, 
given our plan to use an Ocatvia UDP package to emit health messages from the 
VMs to the controller and since controller provision VMs in Nova it might be a 
better integration point for appliances to have custom controllers. I am just 
not convinced that a custom driver is sufficient for all cases –


From: Stephen Balukoff [mailto:sbaluk...@bluebox.net]
Sent: Wednesday, September 03, 2014 10:01 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Octavia] Question about where to render haproxy 


At today's Octavia meeting, one of the questions briefly discussed was where, 
in the Octavia architecture, haproxy configuration files should be rendered. 
This discussion went long and we decided to take it to the mailing list for 
more thorough discussion and to gather alternate ideas.

Anyway, the two main ideas are to render the configuration in the back-end 
(haproxy) driver, and push complete configs out to the VMs/containers running 
the haproxy software via API, or to use the back-end API to push out 
configuration updates and have the VMs/containers render them themselves.

I'm in favor of rendering haproxy configs in the driver, and will present 
arguments in favor of this architecture. I understand German @ HP is the main 
proponent of rendering the configs on the back-end VMs/containers and will let 
him speak to his own points there, and respond to mine below.

Why we should render haproxy configurations within the driver:

  *   This is analogous to how other back-end drivers for proprietary virtual 
machines will do it. This means our reference implementation is easier used as 
a model for 3rd party vendors who want to use Octavia for managing their own 
appliance images.
  *   This keeps the back-end API simpler, as the back-end Octavia VM / 
container doesn't need to know anything about how to render a configuration, it 
just needs to know how to run it.
  *   Simpler back-end API means fewer failure scenarios to have to plan for. 
Either the config pushed out works or it doesn't.
  *   Minor bugfixes, configuration-related security fixes, and minor feature 
improvements can be done centrally without having to update potentially 10's of 
thousands of back-end VMs/containers. (The alternative is to have to do this 
for even the smallest of updates using the other model.)
  *   Major bugfixes and changes will still need to touch all back-end 
VMs/containers, but this is no different than the other model.
  *   If an operator wants to deliver service using another set of load 
balancing software (eg. nginx) this can be done either by writing a new driver 
for a new VM / container image which does this, or by "enhancing" the haproxy 
driver to be able to render both configs and updating the image and back-end 
API to know how to deal with nginx configs. No real advantage or disadvantage 
here, IMO.
  *   This better follows the "centralize intelligence / decentralize workload" 
development philosophy for the project. Putting the rendering logic on the 
Octavia VMs / containers unnecessarily pushes a fair bit of intelligence out to 
what should be fairly "dumb" workload-handling instances.
  *   It is far simpler for the driver, being part of the controller, to pull 
in extra data needed for a complete config (eg. TLS certificates) than for the 
back-end VM / container to do this directly. The back-end VM / container should 
not have privileged resource access to get this kind of information directly 
since that could easily lead to security problems with the whole of OpenStack.
  *   Since the Octavia VM / container image is "dumber" it's also simpler to 
write, test, and troubleshoot.
  *   Spinning up new instances is far simpler because the new back-end just 
gets a complete config from the driver / controller and runs with it. No need 
to try to guess state.
  *   This design is, overall, more idempotent.  (Something went wrong with 
that update to a pool member configuration? No worries-- the next update will 
get that change, too.)
The only down-side I see to this is:

  *   We will need to keep track of which VMs / containers are running which 
images, and the driver will need to know how to speak to them appropriately.  
However, I think this is a false down-side, since:

     *   In a large installation, it's unreasonable to expect the potentially 
10's of thousands of VMs to be running the exact same version of software. 
Forcing this requirement makes things very inflexible and a lot more risky for 
the operator, as far as maintenance and upgrade schedules are concerned.
     *   The above point is to say, having the ability to run with different 
versions of back-end VMs / containers is actually a good thing, and probably a 
requirement for large installations anyway.
     *   If you really want the model where you have to update all your 
back-end VM / container images with each update, you can still do that with 
this topology. Again, with the alternate topology suggested you are forced to 
do that even for very minor updates.
     *   API versioning (even for the back-end) is also a requirement of this 
project's philosophy. And if you're going to have a versioned API:
     *   It's pretty simple to write a 'status' API command which among other 
things lists the API version in use, as well as the versions of all the major 
utilities installed (eg. haproxy, openssl, nginx, etc.) It's then up to the 
controller to render configurations appropriate to the software versions 
installed and/or to force an image to be retired in favor of spinning up a new 
one and/or return an error to the user trying to push out a feature request for 
an image that doesn't support it. Lots of options, all of which can be 
intelligently handled within the driver / controller.


Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
OpenStack-dev mailing list

Reply via email to