Re: Dynamic configuration of workers
You are my hero. The first feature on mod_cluster's webpage I exactly what I'm looking for. Thank you. I'll report here after I've had some time to check it out. -Mark On May 9, 2010, at 2:53 AM, "Mark Thomas" wrote: > On 07/05/2010 23:20, Smith, Mark wrote: >>> -Original Message- >>> From: Pid [mailto:p...@pidster.com] >>> There are a couple of linux load balancer projects that might >>> work, if >>> you can ditch HTTPD. E.g. www.linuxvirtualserver.org >> >> We use LVS to balance load across our Apache layers already, so I'm >> quite familiar with it. >> >> It does do what I want here, except that there are several things >> we need from Apache: Access control, SSL termination, URL path >> based routing, etc. >> >> I considered using ipvsadm on the Apache box to route traffic to >> TomCats, but there is another EC2 specific problem: All the >> routing methods LVS uses don't work on EC2 because they _ONLY_ >> route TCP, UDP and ICMP: >> - Direct Routing messes with Ethernet headers. Not a chance. >> - Tunneling uses IP-in-IP tunneling, which is neither TCP, UDP nor >> ICMP. *grump* >> - NAT gets blocked by the EC2 firewalls, which makes sense. >> - I even tried setting up GRE tunnels; no love. >> >> So, yeah. Thought of that already too. :-) >> >> Anyone else have any ideas? So far, modifying /etc/hosts looks >> like the best solution, even though it tips my kludge-o-meter past >> my comfort zone. > > Take a look at mod_cluster from JBoss. I haven't looked at it for a > while but when I saw it at ApacheCon EU last year it looked like it > might be what you are looking for. > > Mark > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Dynamic configuration of workers
Mark Thomas wrote: On 07/05/2010 23:20, Smith, Mark wrote: -Original Message- From: Pid [mailto:p...@pidster.com] There are a couple of linux load balancer projects that might work, if you can ditch HTTPD. E.g. www.linuxvirtualserver.org We use LVS to balance load across our Apache layers already, so I'm quite familiar with it. It does do what I want here, except that there are several things we need from Apache: Access control, SSL termination, URL path based routing, etc. I considered using ipvsadm on the Apache box to route traffic to TomCats, but there is another EC2 specific problem: All the routing methods LVS uses don't work on EC2 because they _ONLY_ route TCP, UDP and ICMP: - Direct Routing messes with Ethernet headers. Not a chance. - Tunneling uses IP-in-IP tunneling, which is neither TCP, UDP nor ICMP. *grump* - NAT gets blocked by the EC2 firewalls, which makes sense. - I even tried setting up GRE tunnels; no love. So, yeah. Thought of that already too. :-) Anyone else have any ideas? So far, modifying /etc/hosts looks like the best solution, even though it tips my kludge-o-meter past my comfort zone. Take a look at mod_cluster from JBoss. I haven't looked at it for a while but when I saw it at ApacheCon EU last year it looked like it might be what you are looking for. Also, modifying /etc/hosts may not be the panacea. I would imagine that e.g. mod_jk instances do not do a name resolving call at each transaction, and that they cache the result. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Dynamic configuration of workers
On 07/05/2010 23:20, Smith, Mark wrote: >> -Original Message- >> From: Pid [mailto:p...@pidster.com] >> There are a couple of linux load balancer projects that might work, if >> you can ditch HTTPD. E.g. www.linuxvirtualserver.org > > We use LVS to balance load across our Apache layers already, so I'm quite > familiar with it. > > It does do what I want here, except that there are several things we need > from Apache: Access control, SSL termination, URL path based routing, etc. > > I considered using ipvsadm on the Apache box to route traffic to TomCats, but > there is another EC2 specific problem: All the routing methods LVS uses > don't work on EC2 because they _ONLY_ route TCP, UDP and ICMP: > - Direct Routing messes with Ethernet headers. Not a chance. > - Tunneling uses IP-in-IP tunneling, which is neither TCP, UDP nor ICMP. > *grump* > - NAT gets blocked by the EC2 firewalls, which makes sense. > - I even tried setting up GRE tunnels; no love. > > So, yeah. Thought of that already too. :-) > > Anyone else have any ideas? So far, modifying /etc/hosts looks like the best > solution, even though it tips my kludge-o-meter past my comfort zone. Take a look at mod_cluster from JBoss. I haven't looked at it for a while but when I saw it at ApacheCon EU last year it looked like it might be what you are looking for. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Dynamic configuration of workers
> -Original Message- > From: Pid [mailto:p...@pidster.com] > There are a couple of linux load balancer projects that might work, if > you can ditch HTTPD. E.g. www.linuxvirtualserver.org We use LVS to balance load across our Apache layers already, so I'm quite familiar with it. It does do what I want here, except that there are several things we need from Apache: Access control, SSL termination, URL path based routing, etc. I considered using ipvsadm on the Apache box to route traffic to TomCats, but there is another EC2 specific problem: All the routing methods LVS uses don't work on EC2 because they _ONLY_ route TCP, UDP and ICMP: - Direct Routing messes with Ethernet headers. Not a chance. - Tunneling uses IP-in-IP tunneling, which is neither TCP, UDP nor ICMP. *grump* - NAT gets blocked by the EC2 firewalls, which makes sense. - I even tried setting up GRE tunnels; no love. So, yeah. Thought of that already too. :-) Anyone else have any ideas? So far, modifying /etc/hosts looks like the best solution, even though it tips my kludge-o-meter past my comfort zone. -Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Dynamic configuration of workers
On 07/05/2010 22:58, Smith, Mark wrote: >> -Original Message- >> From: Pid [mailto:p...@pidster.com] >>> ...isn't that what I just described? Or is there a part to your >>> suggestion I missed? >> >> It is. :) >> >> I hadn't grokked the whole thread before I started writing the answer >> and missed the last bit. Am on a v small screen. Doh. > > No worries. Just making sure I didn't miss something. Thanks for the help! > :-) There are a couple of linux load balancer projects that might work, if you can ditch HTTPD. E.g. www.linuxvirtualserver.org p signature.asc Description: OpenPGP digital signature
RE: Dynamic configuration of workers
> -Original Message- > From: Pid [mailto:p...@pidster.com] > > Another option is to configure mod_jk for static hostnames then use > > either /etc/hosts or an actual DNS cluster to change the mappings. > > This relies on the application in question honoring DNS TTLs (hint: > > Java doesn't by default). The good news is, it appears that > > mod_proxy_balancer at least honors /etc/hosts. > > > > This is actually my current favorite option, but it just seems like a > > grand hack, and it doesn't allow me to _ADD_ hosts to the config, only > > change them. So I'm likely to configure a large number of workers who > > will not be running most of the time, let the auto-scaling system make > > the workers work or not-work as it brings instances up and down, and > > accept the restart of Apache when I have to increase the max number of > > workers. > > > > It's not pretty, but it looks like it'll work. > > Cheap & dirty. > > 1. pre-configure each load balancer with a list of disabled workers: > > > BalancerMember http://tomcat01.mynet.internal:8009 > BalancerMember http://tomcat02.mynet.internal:8009 > BalancerMember http://tomcat03.mynet.internal:8009 > > # ... > BalancerMember http://tomcat99.mynet.internal:8009 status=D > > 2. update the hosts file on the load balancer to assign your newly > created virtual server IPs to one of the fake domain names above > > 3. enable the no-longer-a-dummy host using the /balancer-manager URLs > > > A variation on this will probably work for mod_jk, it might even work > better. ...isn't that what I just described? Or is there a part to your suggestion I missed? -Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Dynamic configuration of workers
On 07/05/2010 21:28, Smith, Mark wrote: >> -Original Message- >> From: Jon Brisbin [mailto:jon.bris...@npcinternational.com] >> On May 7, 2010, at 1:16 PM, Smith, Mark wrote: >>> Is there a way to tell mod_jk that it has a new worker on a new >>> hostname without having to reload Apache? Similarly, to remove an >>> existing worker that has gone away, or just to change the hostname of a >>> worker. I'm picturing an API on the status worker, or something >>> similar. >>> >>> I don't want to just rewrite workers.properties and do a reload >>> because we have several long uploads happening at any given time that >>> get interrupted. Reload/graceful works most of the time, but >>> occasionally locks up the entire Apache instance, which is no good. >>> >>> I can't imagine we're the first to run into this problem, but I've >>> done a lot of looking and can't find anything other than "rewrite >>> workers.properties and reload Apache." >> >> Reconfig/reload is essentially what I'm doing too, though I use >> mod_proxy_ajp rather than mod_jk. >> >> But the annoying thing is that you're absolutely right you're not the >> first and it's frustrating knowing it'll be a while until enough >> developers have a handle on cloud architectures to make meaningful >> contributions to projects to address these fundamental problems. > > Ok, so the silver lining here is that I'm not a complete buffoon who just > doesn't know how to use Google to find the solution; there are, in fact, > others with the same problem. That's a plus... :-) > >> [...] To be honest, I'm seriously looking at using >> ipchains or something to do NAT on the box and cut out Apache entirely. > > I thought about this. The problem is that you want DNAT to change the > destination of your outgoing packets, but DNAT is only a valid target in > PREROUTING which only gets hit when receiving packets. SNAT is valid in > POSTROUTING on the outgoing side, but doesn't help you with this problem. > > I have this at the top of all my IPTables files; feel free to use it. > > - snip! - > # > # For Reference > # > # Packets traversal through netfilter: > # > #in -> (nat ) -> [routing ] -> (FORWARDING) -> (nat) -> out > # (PREROUTING)[decision](POSTROUTING) > # || /\ > # \/ || > #(INPUT) (nat OUTPUT) > # || /\ > # \/ || > # [local ] -> (OUTPUT) > # [process] > - snip! - > > Another option is to configure mod_jk for static hostnames then use either > /etc/hosts or an actual DNS cluster to change the mappings. This relies on > the application in question honoring DNS TTLs (hint: Java doesn't by > default). The good news is, it appears that mod_proxy_balancer at least > honors /etc/hosts. > > This is actually my current favorite option, but it just seems like a grand > hack, and it doesn't allow me to _ADD_ hosts to the config, only change them. > So I'm likely to configure a large number of workers who will not be running > most of the time, let the auto-scaling system make the workers work or > not-work as it brings instances up and down, and accept the restart of Apache > when I have to increase the max number of workers. > > It's not pretty, but it looks like it'll work. > >> Or even writing my own proxy server from scratch. I hate that I keep >> coming back to that so often lately. I just get frustrated, throw up my >> hands, and resign myself to simply doing it myself. I lamented on this >> very topic yesterday to The 451 Group when we talked about cloud >> architectures etc... > > I guess that's why we all like Open Source: Code doesn't do what you want it > to? Make it do it! ;-) > > I've considered modifying mod_jk or mod_proxy_balancer to add this > functionality, but don't have the resources to do that just yet. Cheap & dirty. 1. pre-configure each load balancer with a list of disabled workers: BalancerMember http://tomcat01.mynet.internal:8009 BalancerMember http://tomcat02.mynet.internal:8009 BalancerMember http://tomcat03.mynet.internal:8009 # ... BalancerMember http://tomcat99.mynet.internal:8009 status=D 2. update the hosts file on the load balancer to assign your newly created virtual server IPs to one of the fake domain names above 3. enable the no-longer-a-dummy host using the /balancer-manager URLs A variation on this will probably work for mod_jk, it might even work better. p > -Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > signature.asc Des
RE: Dynamic configuration of workers
> -Original Message- > From: Jon Brisbin [mailto:jon.bris...@npcinternational.com] > On May 7, 2010, at 1:16 PM, Smith, Mark wrote: > > Is there a way to tell mod_jk that it has a new worker on a new > > hostname without having to reload Apache? Similarly, to remove an > > existing worker that has gone away, or just to change the hostname of a > > worker. I'm picturing an API on the status worker, or something > > similar. > > > > I don't want to just rewrite workers.properties and do a reload > > because we have several long uploads happening at any given time that > > get interrupted. Reload/graceful works most of the time, but > > occasionally locks up the entire Apache instance, which is no good. > > > > I can't imagine we're the first to run into this problem, but I've > > done a lot of looking and can't find anything other than "rewrite > > workers.properties and reload Apache." > > Reconfig/reload is essentially what I'm doing too, though I use > mod_proxy_ajp rather than mod_jk. > > But the annoying thing is that you're absolutely right you're not the > first and it's frustrating knowing it'll be a while until enough > developers have a handle on cloud architectures to make meaningful > contributions to projects to address these fundamental problems. Ok, so the silver lining here is that I'm not a complete buffoon who just doesn't know how to use Google to find the solution; there are, in fact, others with the same problem. That's a plus... :-) > [...] To be honest, I'm seriously looking at using > ipchains or something to do NAT on the box and cut out Apache entirely. I thought about this. The problem is that you want DNAT to change the destination of your outgoing packets, but DNAT is only a valid target in PREROUTING which only gets hit when receiving packets. SNAT is valid in POSTROUTING on the outgoing side, but doesn't help you with this problem. I have this at the top of all my IPTables files; feel free to use it. - snip! - # # For Reference # # Packets traversal through netfilter: # #in -> (nat ) -> [routing ] -> (FORWARDING) -> (nat) -> out # (PREROUTING)[decision](POSTROUTING) # || /\ # \/ || #(INPUT) (nat OUTPUT) # || /\ # \/ || # [local ] -> (OUTPUT) # [process] - snip! - Another option is to configure mod_jk for static hostnames then use either /etc/hosts or an actual DNS cluster to change the mappings. This relies on the application in question honoring DNS TTLs (hint: Java doesn't by default). The good news is, it appears that mod_proxy_balancer at least honors /etc/hosts. This is actually my current favorite option, but it just seems like a grand hack, and it doesn't allow me to _ADD_ hosts to the config, only change them. So I'm likely to configure a large number of workers who will not be running most of the time, let the auto-scaling system make the workers work or not-work as it brings instances up and down, and accept the restart of Apache when I have to increase the max number of workers. It's not pretty, but it looks like it'll work. > Or even writing my own proxy server from scratch. I hate that I keep > coming back to that so often lately. I just get frustrated, throw up my > hands, and resign myself to simply doing it myself. I lamented on this > very topic yesterday to The 451 Group when we talked about cloud > architectures etc... I guess that's why we all like Open Source: Code doesn't do what you want it to? Make it do it! ;-) I've considered modifying mod_jk or mod_proxy_balancer to add this functionality, but don't have the resources to do that just yet. -Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Dynamic configuration of workers
On May 7, 2010, at 1:16 PM, Smith, Mark wrote: > I'm trying to re-architect our websites to work in EC2. One of the biggest > problems I'm running into is the dynamic nature of hostnames and IPs. I have the same problem on a VMware vSphere-based virtual private cloud... > Is there a way to tell mod_jk that it has a new worker on a new hostname > without having to reload Apache? Similarly, to remove an existing worker > that has gone away, or just to change the hostname of a worker. I'm > picturing an API on the status worker, or something similar. > > I don't want to just rewrite workers.properties and do a reload because we > have several long uploads happening at any given time that get interrupted. > Reload/graceful works most of the time, but occasionally locks up the entire > Apache instance, which is no good. > > I can't imagine we're the first to run into this problem, but I've done a lot > of looking and can't find anything other than "rewrite workers.properties and > reload Apache." Reconfig/reload is essentially what I'm doing too, though I use mod_proxy_ajp rather than mod_jk. But the annoying thing is that you're absolutely right you're not the first and it's frustrating knowing it'll be a while until enough developers have a handle on cloud architectures to make meaningful contributions to projects to address these fundamental problems. I'm having to write a lot of stuff myself because almost no software out there that's designed to run web applications addresses the real needs of heterogenous instances of servers that need to work together in a very loosely-coupled and dynamic environment. Tomcat and Apache both have serious shortcomings when it comes to cloud architectures. They're getting better (slowly), but it'll still be a year or more before there's a critical mass of software and experience that make deploying into the hybrid cloud easier. At the bare minimum, I wish the Apache load balancer had a REST API that could add and remove balancer members on the fly, like what you've mentioned above. This single thing would solve the VAST majority of my load balancer problems. To be honest, I'm seriously looking at using ipchains or something to do NAT on the box and cut out Apache entirely. Or even writing my own proxy server from scratch. I hate that I keep coming back to that so often lately. I just get frustrated, throw up my hands, and resign myself to simply doing it myself. I lamented on this very topic yesterday to The 451 Group when we talked about cloud architectures etc... Sort of related: I just put some serious updates into my vcloud session manager that uses RabbitMQ to do non-multicast session "clustering" in a cloud environment, which is the next problem I had to tackle after getting past my load balancer... :) http://github.com/jbrisbin/vcloud/tree/master/session-manager/ Jon Brisbin Portal Webmaster NPC International, Inc. > > Current versions: > - Base system: RedHat EL4 (though we are not using their TomCat) > - Apache: v2.0.63 (grabbed RedHat's EL4 2.0.52 SRPM and up-ported it to > 2.0.63, and added a couple patches of our own for various security reasons.) > - tomcat-connectors: v1.2.25 > - tomcat: v5.5.17 > > I'm certainly willing to consider alternate versions if they'll get me the > functionality I'm looking for. > > Thanks in advance for your help. > > -Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Dynamic configuration of workers
I'm trying to re-architect our websites to work in EC2. One of the biggest problems I'm running into is the dynamic nature of hostnames and IPs. Is there a way to tell mod_jk that it has a new worker on a new hostname without having to reload Apache? Similarly, to remove an existing worker that has gone away, or just to change the hostname of a worker. I'm picturing an API on the status worker, or something similar. I don't want to just rewrite workers.properties and do a reload because we have several long uploads happening at any given time that get interrupted. Reload/graceful works most of the time, but occasionally locks up the entire Apache instance, which is no good. I can't imagine we're the first to run into this problem, but I've done a lot of looking and can't find anything other than "rewrite workers.properties and reload Apache." Current versions: - Base system: RedHat EL4 (though we are not using their TomCat) - Apache: v2.0.63 (grabbed RedHat's EL4 2.0.52 SRPM and up-ported it to 2.0.63, and added a couple patches of our own for various security reasons.) - tomcat-connectors: v1.2.25 - tomcat: v5.5.17 I'm certainly willing to consider alternate versions if they'll get me the functionality I'm looking for. Thanks in advance for your help. -Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
