Hi,
First, that you, Willie and everyone else who has contributed to this
tool that we use heavily in production.
I can imagine quite a lot of work under the covers had to happen to
enable this. Here is what I am thinking of and some background about it.
Obviously, we are just one use case. Please view this as just some ideas
to help seed the discussion.
We are building a run-time orchestrater (honcho) that will manage the
running apps inside a kubernetes pod. We have large complex apps and we
often need to do various things that don't fit the typical "docker run"
model. Honcho also does all the monitoring of the app, so we will be
using the agent addr/port hooks for each host so that haproxy talks to
honcho for health checks but feeds connections to the app address/port.
(thanks for having the flexibility to do this.) When kubernetes starts
up an instance of an app, either a kubernetes hook or honcho will do a
dynamic DNS insert of the records into the zone for the data center.
TXT records are always a hack, but its the way I came up with to allow
per target information to get into haproxy from DNS.
We use lots of alternate ports as well as the regulars, so we would
expect to use the nonstandard _portnum._tcp.name form as well as the
_https._tcp standard form. A couple examples of the names I expect to use:
_12354._tcp.service1.datacenter.soundhound.com
_https._tcp.web2.datacenter.soundhound.com
The entire record set might look like:
($origin datacenter.soundhound.com)
_12354._tcp.service1 in SRV 10 50 23465 target1.service1
target1.service1 in A 10.1.2.3
t1honcho.service1 in A 10.1.2.2
target1.service1._12354._tcp.service1 in TXT "addr t1honcho.service1"
"port 19999"
The config might look like this
frontend service1fe
bind *.12354
default_backend service1be
backend service1be
txt_labels 2 # prepend this many labels from the name of the target
to look for txt records
server _12354._tcp.service1
You could even just do a one liner, since the srv record name specifies
what to list on and the targets.
listen xxx
srv _12354._tcp.service1
This would assume that you are listening on all interfaces...
One last thing. We want to be able to make it so as soon as the DNS zone
is updated, haproxy rereads the DNS for these. DNS has a feature called
notify where a zone master can tell the slaves to do zone transfer so
you can minimize the skew across a set of nameservers.
As a simple first step, I can write a tiny python app that can receive
DNS notify messages when a zone is updated. It would then want to kick
haproxy to reread the DNS. I don't know haproxy well, there may already
be a signal or message that exists that can do this. It would be a small
"nice to have" to teach haproxy to accept notify messages and update
automatically. (You don't need to do the zone transfer, you just need to
receive the message for the new SOA and reply with the correct response.)
I hope all this helps,
jerry
On 12/19/16 4:15 AM, Baptiste wrote:
Hi Jerry,
Thanks a lot for your email and your nice feedback!
When checking SRV records, I first did not really know what to do with
the "priority" field, then I though we could use it to know which set
of servers to use first (as for the 'backup' keyword), exactly as you
mentioned!
Before being able to add SRV records in HAProxy, we had to had more
flexibility (IE: ability to change a server's port...). Now,
everything seems to be ready for SRV records and we can move forward
on the design.
I have an etherpad running somewhere where I'll summarize this design
and share it with the people interested (you, Conrad, Pavlos, HAProxy
Technologies devs, etc...). So we can modify it live, find an
agreement and let the people who want to code it, code it.
This will be "phase 1", using SRV records to update a unique server.
Then, we'll have to work on phase 2: use a DNS query to fill up
servers in the backend. As Willy explained already, there will be a
discussion on the ML too, because the design is not that simple and we
expect the people with experience like you to help us on this stage!
Baptiste
On Mon, Dec 19, 2016 at 6:29 AM, jerry <[email protected]
<mailto:[email protected]>> wrote:
Igor,
thanks for the pointer tot he email thread. We have too many
pieces of automation going on and adding consul really isn't an
option.
Baptiste, if you are open to discussing some requirements I would
love to talk. I am an old hand at DNS and have some specific ideas
of how it could work with haproxy, If you folks are working toward
using SRV records as a way of specifying the current backend
server set, I will take that as a significant opinion that this is
not a bad idea.
thanks again,
jerry
On 2016-12-18 8:45 pm, Igor Cicimov wrote:
On Mon, Dec 19, 2016 at 11:43 AM, jerry <[email protected]
<mailto:[email protected]>> wrote:
Hi,
We use haproxy quite a but at soundhound for fronting our
various external and internal services. We are i the process
of moving to a container based deployment model. With
kubernetes in particular, it's as easy as editing 1 line to
add a new server to a replication set. In one of our
applications, we have 50+ partitions, so when a new instance
is added, it is actually 50+ containers and 50+ proxy
configs to edit. Clearly we need to automate this more than
we have so far.
I was looking around at how to do this and came up with an
unusual idea. Everybody uses DNS, but almost no one
uses/knows about SRV records. SRV records have the form of
_(service)._(protocol).name and return a target DNS name
(like CNAME), priority, weight and destination port. This is
like MX records on steroids.
For SRV records specifically, I'm using Consul and dnsmasq so on
the haproxy instances I can do:
$ dig +short srv tomcat.service.consul
1 1 8080 ip-192.168-0-45.node.dc-prod.consul.
1 1 8080 ip-192.168-1-226.node.dc-prod.consul.
and update the appropriate backend section of the config file and
reload in case of changes. I keep the config file "modular"
meaning multiple parts (mostly for the backends) that I can
update based on some criteria and then glue together to create
the final haproxy.cfg. How do you implement the updates, the
triggers and the config file setup depends on your architecture
and needs.
If you google for "consul-template + haproxy" I'm sure you will
also come across several solutions.
The dynamic DNS update is coming to haproxy in version 1.7, check
this discussion for details
https://www.mail-archive.com/[email protected]/msg23579.html
<https://www.mail-archive.com/[email protected]/msg23579.html>
This seems to have everything I need for the dynamic parts of
a configuration. The original name defines the port to listen
on (this can be a name that gets looked up in /etc/services
or _31112 for a hard port number). The set of records
returned is the backend set. The ones that have the lowest
priority are active and the rest are backups. If you want to
do per backend commands, those can be put in as TXT records
for the name backend.system.name._(service)._(protocol).name.
Similarly, dynamic parts of the frontend and backend could be
done with TXT records for _frontend and _backend prefixes to
the SRV name.
For now, we can assume that there is only one level of backup
rather than the arbitrary number that DNS allows.
There is an extra little bit that DNS can do for us. DNS has
a notify capability. When a zone updates, the dns server can
notify it's slaves with the new SOA record. It turns out that
all you need to do is generate a response that is almost an
exact copy of the query you got, in DNS wire form to make DNS
happy and then do the lookups again.
My thought is that the typical thing you configure for a
given proxy would remain in the config file as now (what to
check, connection mode...). The front end becomes the SRV
name and the backend is either empty of have some command
that says expand the SRV record and put it as the backends.
The reason I like it is there are 75 different
orchestration/automation systems that could want to drive
haproxy. They all talk to DNS already. With this, there is a
single interface that I need for the automation from the view
of both an orchestration system and haproxy. We have a custom
load balancer that we plan to drive the same way.
Here are my questions:
Do people think this is a interesting/good idea?
Is there anything that is missing and would make this a
terrible idea?
If not fatally flawed, should this be run outside of haproxy
as an automation tool or should haproxy do this itself?
thanks,
jerry
--
Igor Cicimov |DevOps
p. +61 (0) 433 078 728
e. [email protected] <http://encompasscorporation.com/>
w*.* www.encompasscorporation.com
<http://www.encompasscorporation.com>
a. Level 4, 65 York Street, Sydney 2000
