Hello modperl-folk: In a nutshell: I would like to ask the community for pointers on how to evolve our successfully deployed application which gets restarted once / hour to reload configuration state to a model where it is continuously running.
Background: We have a long-successfully-deployed mp2 application that has to scale well. Our clusters are currently fronted by hardware load balancers. To effect a configuration change, which is currently done hourly, a process removes a cluster slave host from the hardware load balancers rotation, pushes the new config state down to the slave, and restarts apache2. After apache2 restarts, the mgmt process puts that host back into the load balancer rotation. We are running the prefork MPM with 32-512 httpd children on fairly beefy linux boxes with 4 8 cores each using the standard unix copy-on-write model: load the config state into perl data structures in the parent once via a PerlPostConfigHandler then fork as many children as the prefork MPM decides it needs to handle the load. The configuration pushes are scheduled via wall clock time; currently it is deterministic when a configuration change will be fully propagated. All configuration changes are pushed to all slaves. There are currently a few hundred slaves covering most of North America. This model has worked and is working fairly well for us to date. However, the clock-work nature of our scheme ultimately limits our scalability. And in thinking about what it would take to overcome our current limitations, I would like to be able to reload the configuration state from new data without an apache restart. Put another way, I would like to be able to load next generation configuration state into new httpd children, then kill off the previous generation as they complete their current requests, and start using the new instances, all while servicing requests, albeit at perhaps a reduced rate while the configuration state is being swapped. I don¹t know how to do this in the prefork MPM; and what I am proposing more-or-less breaks the load-parent / fork model. The size of the configuration data are not too large: lets say 10Meg or so, but large enough that I prefer models that share this state among all httpd children. At this point, I'd be willing to consider other MPMs if they help me get there. I am willing to trade memory and CPU to achieve a non-stop apache instance. So modperl experts: any pointers on prior art here? I'd love to hear about strategies that have been shown to work in real life. TIA, -- jeff
