I understand your point. The fact is that I am running on a very small budget. I need the site to scale, but I also need to only use as few servers as possible (Amazon EC2 instances aren't that cheap unless I can minimize the size and number of instances used). Although my budget is small, I also have a requirement for High Availability. Recent outages for Amazon has me thinking I need to spread servers into multiple zones and even different regions. This all adds up to a higher Amazon bill.
Anyway, I have so much work to do in planning and setting up this site that I just resist adding new tasks to my list. I would rather have HAProxy directly support more dynamic configuration than to explore other tools that can automate dynamic reconfiguration of HAProxy. Amazon already provides CloudWatch and Auto-Scale to manage spinning up new servers or taking them down. But, unless I use Amazon's ELB instead of HAProxy, there isn't really built in support for dynamic HAProxy configuration. I'd rather use HAProxy than ELB since I plan on moving to colocation when my Amazon bill goes over $500 a month. On Jan 9, 2013, at 5:57 PM, Zachary Stern <[email protected]> wrote: > If you need this kind of functionality, you are probably running some kind of > large infrastructure right? Or at least a lot of webservers or backend > servers. You would do well to look into some automation there. There are > plenty of existing tools. > > > On Wed, Jan 9, 2013 at 5:47 PM, Kevin Heatwole <[email protected]> wrote: > You might be right that the best way to do dynamic configuration is to have a > tool from a third-party (or created in house) that does monitoring of the > backend servers and edits the config file itself and reloads haproxy. > > I just don't want the hassle of finding such tools or writing my own. Maybe > haproxy could contain such a tool that is separate from haproxy but > maintained, tested, and delivered with haproxy. I have to admit that I > haven't researched any of the tools you use. I should do that, but I always > worry about creating a new dependency on a new tool since these tools are > usually available for free and could lose support and maintenance at any time. > > HAProxy already has some support for dynamic configuration in the health > checks that mark down/up a server depending on the result of the health > check. I figure it is relatively simple to build configuration checks on top > of the health checks since most of the hard work has already been done for > health checks. > > On Jan 9, 2013, at 5:34 PM, Zachary Stern <[email protected]> wrote: > >> Right, and my point is that you can make it dynamic without changing the way >> haproxy itself works. What your asking for seems like making haproxy itself >> overcomplicated, especially for people with simple deployments. But hey, >> maybe I'm 100% wrong. In fact, let's operate on that assumption. >> >> >> On Wed, Jan 9, 2013 at 5:26 PM, Kevin Heatwole <[email protected]> wrote: >> I guess I wasn't clear again. I'm not talking about "editing" the >> configuration file and reloading HAProxy. >> >> My suggestion is simply to implement a dynamic interface to the backend >> servers so they can change the current behavior of the HAProxy instance >> (especially in a load balanced HAProxy backend). >> >> I'll leave it to the developers to figure out what can be dynamically >> changed and if adding a server to a backend is too complex, then that won't >> be part of the interface. >> >> On Jan 9, 2013, at 5:18 PM, Zachary Stern <[email protected]> wrote: >> >>> I understood completely KT. It's perfectly possible to add new lines to the >>> haproxy config dynamically and automatically using things like puppet. >>> >>> E.g. my iptables configurations are dyanmically generated as I spin up new >>> servers, using puppet and the rackspace API. You could do something >>> similar, regardless of cloud or not. >>> >>> When I spin up a new server, it's connected to puppet, tagged as a certain >>> kind of server, and dynamically added as a backend to haproxy if >>> appropriate. >>> >>> >>> On Wed, Jan 9, 2013 at 5:16 PM, KT Walrus <[email protected]> wrote: >>> I think you might have misunderstood. By "adding new server", I mean to >>> add it as a server in HAProxy configuration. That is, the effect is to add >>> the "server" line for the new server into the config file. This has >>> nothing to do with launching the server in the cloud. It is the reverse of >>> marking a server DOWN, except that the server being marked UP was not >>> originally included in the list of servers for the HAProxy backend. >>> >>> On Jan 9, 2013, at 4:21 PM, Zachary Stern <[email protected]> wrote: >>> >>>> >>>> >>>> On Wed, Jan 9, 2013 at 4:13 PM, Kevin Heatwole <[email protected]> wrote: >>>> 4. Adding new server to backend by having configuration check return new >>>> server configuration. >>>> >>>> I don't know about the other features, but this one I think violates the >>>> UNIX philosophy of "do one thing and do it well". There are already plenty >>>> of tools you can use to achieve this with HAproxy, like puppet or chef, >>>> and things like the ruby fog gem for cloud provisioning, etc. >>>> >>>> >>>> -- >>>> >>>> zachary alex stern I systems architect >>>> >>>> o: 212.363.1654 x106 | f: 212.202.6488 | [email protected] >>>> >>>> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >>>> >>>> www.enternewmedia.com >>>> >>> >>> >>> >>> >>> -- >>> >>> zachary alex stern I systems architect >>> >>> o: 212.363.1654 x106 | f: 212.202.6488 | [email protected] >>> >>> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >>> >>> www.enternewmedia.com >>> >> >> >> >> >> -- >> >> zachary alex stern I systems architect >> >> o: 212.363.1654 x106 | f: 212.202.6488 | [email protected] >> >> 60-62 e. 11th street, 4th floor | new york, ny | 10003 >> >> www.enternewmedia.com >> > > > > > -- > > zachary alex stern I systems architect > > o: 212.363.1654 x106 | f: 212.202.6488 | [email protected] > > 60-62 e. 11th street, 4th floor | new york, ny | 10003 > > www.enternewmedia.com >
