Re: dtrace patches.
On Dec 4, 2008, at 10:52 PM, Paul Querna wrote: Theo Schlossnagle wrote: This is just a followup to a previous mail: http://mail-archives.apache.org/mod_mbox/httpd-dev/200805.mbox/[EMAIL PROTECTED] The apr-utils include mod got immediate uptake. Thanks Paul! I didn't see much progress on the rest of it. How do I got about guiding through the patch acceptance process? Or am I dense and it was indeed committed and I'm just looking in the wrong place. Thanks! I've committed most of your patch to trunk now. I did not however include the bits modifying the build, they seemed a little too creative and didn't work right on my mac -- I'll try to get something fixed up soon, but patches welcome. (This means it doesn't build by default, and it fails to compile right if you enable it, but I think I can get it fixed before cutting the 2.3.0-alpha this weekend). Thanks, Paul Awesome news! Deep thanks for your time investment on this! Some cool possibilities are coming with that changeset. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/
dtrace patches.
This is just a followup to a previous mail: http://mail-archives.apache.org/mod_mbox/httpd-dev/200805.mbox/[EMAIL PROTECTED] The apr-utils include mod got immediate uptake. Thanks Paul! I didn't see much progress on the rest of it. How do I got about guiding through the patch acceptance process? Or am I dense and it was indeed committed and I'm just looking in the wrong place. Thanks! Keep me on the CC, I'm not on list. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/
Re: [PATCH] DTrace probes patch.
On May 6, 2008, at 8:21 AM, Dirk-Willem van Gulik wrote: On May 5, 2008, at 9:27 PM, Theo Schlossnagle wrote: The patch has some nastiness to it that I'm sure people would want to strategize on cleaning up. The main issue being that Apache is constructed from a bunch of static apr/libtool built libraries. DTrace doesn't work on archives. So, I've got some bloody knuckles from bending the build system to keep things as normal ELF objects. I had a first good step... and then a red herring issue that I worked through with the DTrace team led me to a much less-elegant way of building. I could revert to the original process (ld -r -o the objects into library-esque packages) as DTrace can work on those. The probes are neatly defined and placed, but the patches to the build system are gruesome. The apr-util patch to the apr_hooks.h is simple and affords some nice probability for future probe uses. Docs on these probes are available here: https://labs.omniti.com/trac/project-dtrace/wiki/ Applications#Apache2.2.x I'm not on this list -- Cc me on pertinent responses please. Works lovely on Solaris with the normal/real libtool - but not with Justin's. On MacOSX I think there is something wrong with the linker. Either MacOS does not need the extra -G or -h is enough on all platforms. What is the downside/penalty for making this a default ? Or should this always be an optional thing - set at ./configure time ? I see no issues with making this the default and having a --disable- dtrace. I can see a reason that someone might wish to turn them off -- thought that someone isn't me. Note, that to get all the apr_hooks linked up (which allows timing hook call-times and classifying system calls by hook name and phase) we need to patch apr_hooks.h. The patch for that changes flow slightly (but not outcome) and has no cost when disabled. Also, the way I wrote that was to use another define to turn that on APR_DTRACE_PROVIDER. So, anyone using apr-util for hooks can enable or disable probes on those hooks with that define. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/
[PATCH] DTrace probes patch.
Hello all, The probes can really give a different perspective on production environments. The patch has some nastiness to it that I'm sure people would want to strategize on cleaning up. The main issue being that Apache is constructed from a bunch of static apr/libtool built libraries. DTrace doesn't work on archives. So, I've got some bloody knuckles from bending the build system to keep things as normal ELF objects. I had a first good step... and then a red herring issue that I worked through with the DTrace team led me to a much less-elegant way of building. I could revert to the original process (ld -r -o the objects into library-esque packages) as DTrace can work on those. The probes are neatly defined and placed, but the patches to the build system are gruesome. The apr-util patch to the apr_hooks.h is simple and affords some nice probability for future probe uses. Docs on these probes are available here: https://labs.omniti.com/trac/project-dtrace/wiki/ Applications#Apache2.2.x I'm not on this list -- Cc me on pertinent responses please. Best regards, Theo -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ apache-2.2.x-probes-p1.patch Description: Binary data apr-util-hook-probes.patch Description: Binary data
Re: Working on some load balancing methods
On Jan 11, 2005, at 9:44 AM, Jim Jagielski wrote: On Jan 11, 2005, at 4:20 AM, Ben Laurie wrote: Justin Erenkrantz wrote: --On Saturday, January 8, 2005 10:43 PM + Ben Laurie [EMAIL PROTECTED] wrote: Errr... mod_backhand? mod_backhand doesn't support Apache 2.x: http://www.backhand.org/mod_backhand/FAQ.shtml#question0 Port it? I think that we can come much further along with extending the lb capability in proxy... For more sophisticated and demanding environments, an external lb mechanism is likely used. So my pers. pref would be to see what can be done in proxy before seeing if mod_backhand even needs to be ported. I don't think that the web server should need to do everything :) Having mod_backhand use mod_proxy isn't very difficult. We implemented that for a client. I don't understand the comment about the web server doing stuff. mod_proxy sits inside apache and adheres to the same limitations do to it architectural position as mod_backhand. mod_backhand already allows for users to write their own arbitrary load balancing decision functions and run them. Perhaps looking at it wouldn't be so bad as it has been around for a while and understands the problems inherent in building a load balancer inside of a web server. One extremely important thing to consider is that Alteons and all their appliance competitors support 100k concurrent SMTP sessions at a _bare minimum_. Apache 2 is hard pressed to do that. So, you need more than one front-end load balancer to share the load balancing over. At this point, the issues of decision making compound dramatically as total knowledge is lost. As the 2+ front-end load balancers are all accepting client-originated traffic, they don't know the decisions that the other peers are making. This results in hard contention problems to solve -- which mod_backhand accounts for. So, if you go implement something I have some advice. Don't speculate on what you think is cool or what worked for you in your specific environment. Alteon and big/ip and foundry, etc. etc. all have several load balancing policies for a _reason_. They all don't work everywhere. There is a lot of theory and research behind this stuff. go read some Sigmetrics issues and make sure you wrap your head around the whole problem before you solve it -- because you will be disappointed with the it you wind up with. // Theo Schlossnagle // Principal Engineer -- http://www.omniti.com/~jesus/ // OmniTI Computer Consulting, Inc. -- http://www.omniti.com/ // Ecelerity: fastest MTA on Earth
Re: Working on some load balancing methods
On Jan 11, 2005, at 11:00 AM, Jim Jagielski wrote: Theo Schlossnagle wrote: Having mod_backhand use mod_proxy isn't very difficult. We implemented that for a client. I don't understand the comment about the web server doing stuff. mod_proxy sits inside apache and adheres to the same limitations do to it architectural position as mod_backhand. Just that just because you *can* implement something in the web server, doesn't mean you *should* or *must*. A web server is part of one's web infrastructure, not its entirety. I absolutely agree. I misunderstood you. It sounded like you were arguing against mod_backhand and for mod_proxy on the basis that the web server isn't the right place to be doing it... As the both live in the web server proper, it seemed an awkward position. If y'all want to build a good load balancer, use APR and stay our of the Apache server proper completely. It would afford you the opportunity to scale better as much of the plumbing within Apache that guides it architectural design can be shed. Graham and I talked about how to make mod_proxy backhand capable for some time. Neither of us had time on our hands to look at it. mod_backhand isn't a proxy framework, it is a decision making framework that allows complex, adaptable decision making functions to be applied and a collaborative manner using the concept of peer-to-peer systems. The fact that the current version of mod_backhand has its own internal proxying code instead of using mod_proxy via the EAPI is simple a testament to its roots (outside of Apache). I am not at all against making mod_proxy a good load balancer. It would, however, be a shame to lose the innovations (if in concepts only) that has already occurred in a similar project. // Theo Schlossnagle // Principal Engineer -- http://www.omniti.com/~jesus/ // OmniTI Computer Consulting, Inc. -- http://www.omniti.com/ // Ecelerity: fastest MTA on Earth
Re: Working on some load balancing methods
On Jan 11, 2005, at 12:23 PM, Jim Jagielski wrote: Paul Querna wrote: -There is an external programe (started like a piped log program) that does active health checking. Every x seconds it checks all the origin servers and records their status in shared memory. This is more or less what mod_backhand does. (it is more passive, waiting for a active server to send out a multicast or an ethernet broadcast saying that it is alive, along with stats.) I've found that in most places, passive is more acceptable than active... But even so, many places don't like these sorts of broadcasts at all. There is a big resistance to agent-based solutions, especially when living on a web server. One reason why the current proxy lb is attractive is that it is quite self-contained, without packets being spewed all over the network. I'm not saying that a mod_backhand-ish solution isn't a good thing to add to our utility belt, but it should not stop the basic lb capability within mod_proxy. It is a trivial change to make mod_backhand not can about announcements as well as not make them. Even more so if it was integrated with mod_proxy more extensively. // Theo Schlossnagle // Principal Engineer -- http://www.omniti.com/~jesus/ // OmniTI Computer Consulting, Inc. -- http://www.omniti.com/ // Ecelerity: fastest MTA on Earth
Re: Working on some load balancing methods
On Jan 7, 2005, at 4:14 PM, Sander Striker wrote: From: Jim Jagielski [mailto:[EMAIL PROTECTED] Sent: Friday, January 07, 2005 8:52 PM To: dev@httpd.apache.org Subject: Working on some load balancing methods I'm currently working on code that extended the lb method within the 2.1/2.2 proxy from what is basically a weighted request count to also be a weighted traffic count (as measured by bytes transferred) and a weighted load count (as measured by response time). The former is further along and the methods will be selectable at runtime... This is definitely a scratch I'm itching, I'm sure you are not the only one with that itch. You are welcome to harvest any plumbing you like from mod_backhand. It does exactly what you want only in apache 1.3.x but before I spend too much (additional) time on it, I'd like some feedback on whether the concept is one we can all get behind. FWIW, I like it. I am also toying with the idea of supporting a CPU load method when the origin servers are Apache via a custom response header... CPU load is tricky. It has observational bias. System load adjusts two slowly. The number of concurrent connections to each machine actually works pretty well as it suggests that the there are that many Apache children (adapt working as needed for Apache 2) working on the box and it lends itself to a current length of the run queue. All of these methods require total knowledge, otherwise you have contention issues and suffer from inaccuracies due to stale information. There are several nice randomized approaches that work well to smooth our spikes due to stale data. mod_backhand uses a simple stackable selection mechanism called candidacy functions that implement all this stuff (optionally as loadable modules) so they cleverness of the load balancing decisions can be decided later. // Theo Schlossnagle // Principal Engineer -- http://www.omniti.com/~jesus/ // OmniTI Computer Consulting, Inc. -- http://www.omniti.com/ // Ecelerity: fastest MTA on Earth
Re: buffered logs + piped logs
Brian Akins wrote: Paul Querna wrote: mod_log_spread: http://www.backhand.org/mod_log_spread/ Unfortunantly, not an option here :( I'm looking for something to implement before the first Tuesday in November... Out of curiosity, why is that not an option? It's used at a lot of _large_ sites. -- // Theo Schlossnagle // Principal Engineer -- http://www.omniti.com/~jesus/ // Postal Engine -- http://www.postalengine.com/ // Ecelerity: fastest MTA on Earth
Re: [proxy] New implementation ready for testing
On Aug 16, 2004, at 6:54 AM, Pier Fumagalli wrote:
On 11 Aug 2004, at 17:14, Mladen Turk wrote:
Hi all,
We've finished the initial development of extended mod_proxy.
Since the development took place at jakata-tomcat-connectors,
the source code can be found under ajp/proxy.
Here is the list of major features added:
3. Added new module proxy_balancer
What's wrong with:
ProxyPassReverse / http://localhost:/
ProxyPreserveHost On
RewriteMap hosts rnd:/opt/apache/conf/tables/hosts.map
RewriteRule ^/(.*) ${hosts:live}/$1 [P,L]
It's in there already, and it works great (I even have a couple of
CGIs reading and building up that table, enabling, disabling and
prioritizing hosts).
http://wiki.apache.org/cocoon/ApacheModProxy
http://wiki.apache.org/cocoon/LoadBalancingWithModProxy
Having a human weigh or prioritize hosts may work well in a small
environment, but it is too much to manage in a large one. Besides,
while random selection is a decent algorithm under certain
circumstances, mod_proxy lacks the insight into the global state of
resources to make truly intelligent resource-based allocation
decisions.
I haven't had the time lately, but you (all) are welcome to dissect the
mod_backhand (Apache 1.3) module and pull the resource collection and
decision making framework out and put in into mod_proxy for Apache 2.
The algorithms therein are flexible, proven and even extensible. We
have seen several large architectures use m_b with great success in
front of even ISS servers and Java application servers of all types.
http://www.backhand.org/mod_backhand/
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: fastest MTA on Earth
Re: Tweaking for Alpha..
On Friday, December 14, 2001, at 11:57 AM, Greg Ames wrote: Dec 13 17:10:03 kernel: httpd(21879): unaligned trap at 0200012dcd00: 039fdadc 2d 0 Dec 13 17:10:05 kernel: httpd(21922): unaligned trap at 0200012dcd00: 039fdadc 2d 0 I get these in perl all the time on my dual ev67 boxes. I tracked it down a problem with liinking gcc binaries against CCC libs. If you use the Compaq C Compiler (which marvelously faster than gcc in both compilation and the resulting binary) to create a binary (mysql/mysqld and all the supporting libraries) and then link against then with gcc (perl DBD::mysql), I find that I get traps all over the place. The REALLY strange thing was that even with a few traps a second, the programs never misbehaved. Now I compile everything with ccc (truly amazing peformance speed up). Of course, I am not bold enough to try a Linux kernel compile with ccc. You didn't make it clear that you were using Linux, and I just assumed from the format of you syslog messages. If I am wrong, perhaps this message will be useful to someone else :-) -- Theo Schlossnagle 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
