Re: lua-apr and mod_lua
On 7/8/11 1:23 PM, Brian McCallister bri...@skife.org wrote: I have not yet had a chance to look into lua-apr ( http://peterodding.com/code/lua/apr/ ) but it might be darned handy re: mod_lua. I looked at it and it seems to not be that great of a fit. It uses global pools all over the place, so it may be hard to integrate with existing httpd-like stuff. -- Brian Akins
Re: mod_lua Filter Hook?
On 6/15/11 4:08 PM, Joachim Zobel jzo...@heute-morgen.de wrote: mod_perl does this pretty good IMHO. The point is auto generation of usable API wrappers. FWIW, and this is just my opinion, but I'm not not 100% sure that having a complete (or near) complete Lua version of the HTTPD (and APR) API is really worth the effort. I've grown to like the very simple way some other web servers have done it. I've also learn to write as much code in Lua as possible and just have the low level glue in C, especially with the jit. For filters, etc, not sure we really need buckets in Lua. Maybe just represent them as a table of buffers or something simple like that. -- Brian Akins
Re: 3.0, the 2011 thread.
On 6/15/11 6:01 PM, Paul Querna p...@querna.org wrote: pocore: For base OS portability and memory pooling system. http://code.google.com/p/pocore/ How does this compare to APR? libuv: Portable, fast, Network IO. (IOCP programming model, brought to Unix) https://github.com/joyent/libuv I've played with it. It's rough - particularly dealing with memory. http-parser: HTTP really broken out to simple callbacks. https://github.com/ry/http-parser I like this one a lot. selene: SSL, redone to better support Async IO. https://github.com/pquerna/selene Haven't had a chance. +1 to the idea. I still like Lua ;) People said I was crazy when I said Lua should be the config and the runtime - now look at node.js -- Brian Akins
Re: mod_lua Filter Hook?
On 6/15/11 6:26 PM, HyperHacker hyperhac...@gmail.com wrote: = I'd been looking forward to mod_lua for a while now expecting it to work similarly to PHP (handle requests, send output without having to worry about how the httpd works). Is that not the case? Brian M. can correct me, but the original intent for mod_lua (nee mod_wombat) was when you needed to get at the internals of Apache, but didn't want to write a full on C module. Like needed nested if's for a rewrite, a strange auth method, or whatever. It was not really meant to be a competitor to php, ruby, python, etc. for application development. -- Brian Akins
Re: 3.0, the 2011 thread.
On 6/15/11 7:40 PM, Colm MacCárthaigh c...@allcosts.net wrote: Imagine that in turn feeding into a set of co-routine filters. That's fundamentally different - it parallelises content generation, but it's really really hard to do in C. Depending on how far you want to push the model, it's not that hard. Obviously you can't do co-routines but just using the current ideas about requests and sub requests, you could easily do the subrequests in parallel. FWIW, nginx can use Lua co-routines to do this and does it natively with SSI's. The code, however, will make you go blind ;) My biggest issue with HTTPD really comes down to connections per OS image. In general, threads suck at this - memory per connection and context switches just kill you. C1M is just not that hard to achieve nowadays. -- Brian Akins
Re: PHP5.3.6
On 4/15/11 6:11 PM, Reindl Harald h.rei...@thelounge.net wrote: Am 15.04.2011 23:01, schrieb Jeff Trawick: FastCGI is a way to get us out of all sorts of dark alleys; let's be sure to keep it in mind as one of the tools to address binary compatibility issues, PHP-centered or not but you can not use .htaccess with cgi .htaccess works just fine with FastCGI. .htaccess is handled by Apache as normal. FastCGI is just a different way to run the PHP interpreter in this case. -- Brian Akins
Re: a plan mod_lua and 2.4
On 3/14/11 7:51 PM, Brian McCallister bri...@skife.org wrote: I think the right path for this is to shrink the core mod_lua module to be the *minimal* set for integration, and pull all the higher level functionality into a non-core module, which uses the core mod_lua to hook into the server. +1 How many people actual run mod_lua (or a derivative/relative) in production? Am I the only one on a real site? FWIW, moving more and more of the code into Lua and having only very thin wrappers in C is a good idea as well. Plays really nicely with luajit, as well. -- Brian Akins
Re: a plan mod_lua and 2.4
On 3/15/11 10:05 AM, Dan Poirier poir...@pobox.com wrote: Maybe if people could embed lua in web pages. FWIW, I don't think the web needs yet another web page generator embedded inside a web server. We apps should be ran by application servers - fastcgi, proxy, etc. But that's just my opinion ;) -- Brian Akins
Re: a plan mod_lua and 2.4
On 3/15/11 2:25 PM, Dan Poirier poir...@pobox.com wrote: I agree too, but the discussion was about whether mod_lua might replace PHP some day, and I think if that's your goal (worthwhile or not) you'd have to provide web page embedding. We never, ever run mod_php, only via fastcgi, anyway. No need to embed php-like functionality inside mod_lua, IMO. -- Brian Akins
Re: RewriteCond integral comparison
On 9/14/10 11:02 AM, William A. Rowe Jr. wr...@rowe-clan.net wrote: It looks like I need to hack together support for integer comparison operations, al la -lt/-gt/-eq, for RewriteCond evaluation. Eww. FWIW, we axed mod_rewrite mostly and just use Lua now. Once you need this type of logic, is it really worth it to invent yet-another-language? Going back under my bridge... -- Brian Akins
Re: C as config
On 6/8/10 6:34 PM, Sean Conner s...@conman.org wrote: If your platform is x86, have you considered testing with LuaJIT? It compiles Lua code directly into x86 code and is a drop-in replacement for lua (just link against libluajit instead of liblua). The few tests I've done have been impressive [1]. From earlier in the thread: On 6/4/10 7:30 PM, Paul Querna p...@querna.org wrote: Are you using LuaJIT 2? The performance numbers its putting up seemed very impressive. Yes and meh... . The issue is, it still running Lua. Which, while fast compared to lots of other languages is still much slower than C. Even when core httpd is doing all the heavy lifting most of the CPU is spent doing Lua-stuff in our case. And not even the interesting stuff... -- Brian Akins
Re: C as config
On 6/8/10 7:21 PM, Graham Leggett minf...@sharp.fm wrote: That said, if your server doesn't have work to do, ie you're just a bit-shifter, then a simple async loop will win hands down. We also found that even for resource intensive tasks -- like rendering a template or something similar -- it's best just to let it run so it can finish and get out of the way. Only real time it seems to make sense to yield is when you need to wait on IO.* Threaded programming just hides the wait because the OS takes care of the yielding for you. In Lua, I did tests using coroutines to hide yielding to an even loop, and that didn't turn out so well. The code looked awesome, but performance was horrible. *with the popularity of RESTful (or RESTish) database-like things, writing an application server using an event loop should be easier. No more blocking database clients. Ultimately, that what I want to do with the Lua. Well, that and kill mod_rewrite and the like ;) -- Brian Akins
Re: C as config
On 6/8/10 8:07 AM, Dan Poirier poir...@pobox.com wrote: That's a shame. I wonder if another embedded language would do better? E.g. mod_perl, mod_python? They've both been around for a while. I wonder why neither has gotten into the server? Lua is leaps and bounds faster than both of those in my experience. -- Brian Akins
Re: C as config
On 6/7/10 9:16 AM, Dan Poirier poir...@pobox.com wrote: Did you profile httpd? I'm wondering if you had a few non-trivial hooks in lua, if it would be a significant part of the CPU consumption, or would it be swamped by the rest of the processing that always goes on for a request? Increasingly, httpd is just there to provide a base to run Lua stuff for us. You'd be surprised how easily problems get solved with a few if's and else's instead of mounds of rewrite rules. At normal load it doesn't really matter - the trade off is worth it (speed vs simplicity). However, as the servers get more and more loaded, Lua takes a rather large percentage of the CPU. And it's the Lua stuff not the part that does work. The constant string hashing, metatable lookups, etc. So, just as a test I converted a large amount of Lua into C. It was easy enough and was a huge improvement. So, just for Paul's benefit - we generate .so's from C ;) -- Brian Akins
Re: C as config
On 6/6/10 8:03 AM, Graham Leggett minf...@sharp.fm wrote: With multi-core architectures, we're finding that humble worker on our commodity hardware is as fast or faster than our load balancers. +1 We soon found that the web servers spends more time doing context switches than actual work. (Or so it seems, probably a slight exaggeration). Imagine httpd with 100k threads!!! Now imagine the same amount of work being done in 2 threads. Not terribly hard to do in an asynchronous framework. -- Brian Akins
Re: C as config
On 6/4/10 10:37 PM, Paul Querna p...@querna.org wrote: The most iteresting thing in this space since VCL was created is the development of LLVM clang. It seems like if you wanted to build this these days, that would be where to start, though including an entire compiler stack does seem both a dainting task, and one that creates many other issues. Yeah, I thought about that as well. However, I already know C ;) A config/runtime language built using llvm could probably be just as fast as writing straight C. The thing I hate about VCL is that it teases you with being C like, but without the power. Maybe I'm just old and crufty, but C is actually easier for me than 90% of the config languages I've seen. -- Brian Akins
C as config
All of you folks who have to answer user questions, go ahead and ready your hate mail :) I've been playing some with Varnish (long story) and lots of people seem to like it. The config language (VCL) is just a thin wrapper on top of C. Heck, you can just write C inline. Also, I do a good bit with Lua -- in httpd and other projects. Every time I run profiles on this stuff, Lua is always 3 out of the top 5 cpu consumers. And lots of it is just the language itself (all the table look-ups) and we highly optimized our method dispatches (using apr_hash, even had a version using gperf). Now this is probably just fine for mere mortals, but it has always bothered me. We always meant to go back and write higher level modules based on the low level Lua bindings, but we never did and we have several non programmers who are happily writing configs in Lua. As a side project several months ago (maybe I mentioned it here??), I wrote a very simple module that basically loaded small C libraries, examined their symbol table and added certain functions to an array for each hook: int fixups(request_rec *r) would get added to an array of functions ran at fixups phase, etc. I never really pursued it very far, but it was fast. Of course you had all the C things to worry about. So, a fellow engineer and I were talking today and convinced ourselves to strip Lua from another project and just use C as the high level language. Lots of the gory implementation was already hidden in C libraries anyway, so it was just a matter of porting the Lua helpers into straight C. It took less than 2 hours. We were sort of shocked at the results - 1/3 the memory usage and 50% better performance (you run out of CPU eventually, no matter what you do). Odd thing was, the code in C for the Config looked eerily close to the Lua. Of course, that may be a good sign that I know nothing about writing Lua... (And yes, we tested with luajit) So, we thought about wouldn't it just be cool if Apache had VCL. Then I thought, I already know C, why do I need to learn a meta-language that just writes C??? Won't someone think of the users! I could here all of the #httpd folks saying. Not a terribly interesting read, but we are seriously considering just using straight C with some helper functions and macros as the config for one of our projects. And, for the record I was wrong in the past - yes, async is the answer... -- Brian Akins
Re: C as config
On 6/4/10 7:30 PM, Paul Querna p...@querna.org wrote: Are you using LuaJIT 2? The performance numbers its putting up seemed very impressive. Yes and meh... -- Brian Akins
Re: Reducing number of mod_lua hook directives
On 5/12/10 11:09 AM, Brian McCallister bri...@skife.org wrote:
Ah, but what happens when there are multiple functions for same hook?
We could get into magic naming and scanning of global scope, but that
really turns my stomach.
No need. It's easy. Think about it in lua, not C.
Something like this for a loader:
handlers = { }
-- mods is an array of modules. Probably passed in via post_config or
something. Hooks is an array of hook names
function load_stuff(mods, hooks)
-- try to load each mod like
local m = require 'modname'
-- now examine m for functions named in hooks
if m.functionname add to my array
end
Then in you runner something like
function hook_runner(r, hook_name)
-- look in handlers for hook_name. It should be an array of functions
-- call them, with r as the single argument until one returns something
-- other than declined
end
Conceptually, we probably don't want different script authors
stomping on each other anyway.
Forcing the use of modules rather than standalone scripts helps this some.
We loose a lot of the efficiencies of lua because of the design of worker,
prefork, event, etc. Lua Strings, memory, etc, per thread -- with lots of
threads - seems wasteful. But that's a larger discussion.
FWIW, I do a lot of lua now. We use it in almost every web server config
now.
--
Brian Akins
Re: Reducing number of mod_lua hook directives
The hook runner itself should just be in lua. Instead of scripts, you write lua modules. If a lua module has a function with the correct names (ie, same as hooks) it uses that: Ie in my module foo function fixups( r ) Would get called at fixups, etc. No need for a state per file, etc. Just have one state per thread. -- Brian Akins
ProxyPreserveHost per dir??
Is there a particular reason why ProxyPreserveHost is per server and not per dir? We have some unusual configs and I'm wondering what secret internals I may be breaking by changing that? The code didn't show anything obvious. -- Brian Akins
Re: State of mod_lua?
On 2/19/10 6:11 PM, Sander Temme scte...@apache.org wrote: Aside from satisfying your own curiosity (which is a perfectly valid reason), why invent another one? Or am I misreading you? Mod_lua (and mod_wombat before it) has some serious architectural issues. We've talked about them on the list before and in person. I'm hoping I can release our version soon, but I say that every year -- Brian Akins
Re: [PATCH] LogLevel refactoring part 1
On 2/5/10 9:39 AM, Dan Poirier poir...@pobox.com wrote: . E.g. maybe today I'd like to see all debug trace from authentication, but tomorrow just see SSL stuff. +1 -- Brian Akins
Re: reopening of logs without restarting
On 1/22/10 2:18 PM, Graham Leggett minf...@sharp.fm wrote: What you're probably after are piped logs: The amount of context switches this can create is almost mind boggling... -- Brian Akins
Re: reopening of logs without restarting
On 1/22/10 2:39 PM, Graham Leggett minf...@sharp.fm wrote: Trying to signal all the processes, all the threads, and all the event loops to close and reopen a file simultaneously simply isn't feasible or accurate enough to be useful. Each process (not thread) has an open filehandle with mod_log_config, correct? -- Brian Akins
Re: Httpd 3.0 or something else
On 11/10/09 6:20 PM, Greg Stein gst...@gmail.com wrote: I'd like to see a few network threads multiplexing all the writing to clients. That's what I meant. I just didn't state it properly. Then take all of *that*, and spread it across several processes for solid uptime, with a master monitor process. And then you have nginx ;) -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 3:08 PM, Greg Stein gst...@gmail.com wrote: 2) If you have 10,000 client connections, and some number of sockets in the system ready for read/write... how do you quickly determine *which* buckets to poll to get those sockets processed? You don't want to poll idle connections/buckets if only one is ready for read/write. Epoll/kqueue/etc. Takes care of that for you. -- Brian Akins
Re: Httpd 3.0 or something else
On 11/10/09 1:56 PM, Greg Stein gst...@gmail.com wrote: But some buckets might be performing gzip or SSL encryption. That consumes CPU within the network thread. You could just run x times CPU cores number of network threads. You can't use more than 100% of a CPU anyway. The model that some of us discussed -- Greg, you may have invented it ;) -- was to have a small pool of acceptor threads (maybe just one) and a pool of worker threads. The acceptor threads accept connections and move them into worker threads - that's it. A single fd is then entirely owned by that worker thread until it (the fd) goes away - network/disk io, gzip, ssl, etc. -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 12:32 AM, Brian McCallister bri...@skife.org wrote: A 3.0, a fundamental architectural shift, would be interesting to discuss, I am not sure there is a ton of value in it, though, to be honest. So I should continue to investigate nginx? ;) FWIW, nginx delivers on its performance promises, but is a horrible hairball of code (my opinion). We (httpd-dev type folks) could do much better - if we just would. (Easy for the guy with no time to say, I know...) -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 12:52 PM, Graham Leggett minf...@sharp.fm wrote: This gives us the option of prefork reliability, and event driven speed, as required by the admin. I think if we try to do both, we will wind up with the worst of both worlds. (Or is it worse??) Blocking/buggy modules should be ran out of process (FactCGI/HTTP/Thrift). -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 1:18 PM, Graham Leggett minf...@sharp.fm wrote: and we know from the same period of experience from others that a pure event driven model is useful for shipping static data and not much further. It works really well for proxy. -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 1:36 PM, Graham Leggett minf...@sharp.fm wrote: It works really well for proxy. Aka static data :) Nah, we proxy to fastcgi php stuff, http java stuff, some horrid HTTP perl stuff, etc (Full disclosure, I wrote the horrid perl stuff.) -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 1:40 PM, Brian Akins brian.ak...@turner.com wrote: On 11/9/09 1:36 PM, Graham Leggett minf...@sharp.fm wrote: It works really well for proxy. Aka static data :) Nah, we proxy to fastcgi php stuff, http java stuff, some horrid HTTP perl stuff, etc (Full disclosure, I wrote the horrid perl stuff.) Replying to my own post: What we discussed some on list some at Apachecon, was having a really good and simple process manager. Mod_fcgid is too much work to configure for mere mortals. If we just had something like: AssociateExternal .php /path/to/my/php-cgi And it did the sensible thing (whether fcgi, http, wscgi, etc.) then all the config is in one place. Obviously, we could have some advanced process management directives. If your app needed some special config stuff, we could easily pass it across somehow. -- Brian Akins
Re: Httpd 3.0 or something else
On 11/9/09 2:06 PM, Greg Stein gst...@gmail.com wrote: These issues are already solved by moving to a Serf core. It is fully asynchronous. Okay that's one convert, any others? ;) That's what Paul and I discussed a lot last week. My ideal httpd 3.0 is: Libev + serf + lua -- Brian Akins
Re: Httpd 3.0 or something else
On 11/5/09 4:30 PM, Graham Dumpleton graham.dumple...@gmail.com wrote: Thoughts? Still digesting, but generally +1 to the entire post. -- Brian Akins
Httpd 3.0 or something else
So, after several conversations at Apachecon and on the list, we still have no real vision of how we want to move ahead with httpd 3.0. Or, if we do, it's not communicated very well. Some have suggested we just create a new server project. Others want to keep hacking away at the current code base. Thoughts? -- Brian Akins
Re: Fastcgi, use something like spawn-fcgi?
On 10/30/09 1:29 PM, Albert Lash albert.l...@docunext.com wrote: But in the end, it is awesome for containing memory leaks and automatically re-spawning fastcgi processes. An external process-manager can do the same. My point was do we really want/need this complexity inside httpd? Also, the config just seems so clunky compared to the proxy config. In closing, I'll add that I'm greatly appreciative of both Apache and mod_fcgid and was thrilled when I learned that mod_fcgid was getting merged into the Apache code base. I was too, but mostly just because mod_fastcgi was horrible in my testing. -- Brian Akins
Fastcgi, use something like spawn-fcgi?
With all the discussion of mod_fcgid lately (I admit, I did not read all the threads, so forgive me if we already discussed this), I was wondering if a different approach would be better. I know at one time there was work on a mod_proxy_fastcgi. The current trend in other webservers seems to do the proxying inside the webserver and using a separate small process manager - something like spawn-fcgi or supervisord. So, we could have configs something like: ProxyPass /myapp fastcgi:///path/to/my.sock max=6 ... And should easily be able to do fasctcgi over tcp, etc. This should, hopefully, simplify the in httpd code. Just thinking... -- Brian Akins
Re: Fastcgi, use something like spawn-fcgi?
On 10/29/09 9:50 AM, Jeff Trawick traw...@gmail.com wrote: There has been some interest expressed on this list from the mod_fcgid user community to manage processes differently (e.g., independent of load); in general, I'd like to see the proxy+process mgr solution support requirements that would otherwise twist mod_fcgid in new directions. The nginx community just recently released some glue between nginx and supervisord that looks interesting. -- Brian Akins
Stating file again in default_handler?
Did we ever decide what to do about this? http://mail-archives.apache.org/mod_mbox/httpd-dev/200801.mbox/%3CC3C353B8.A 60a1%25brian.ak...@turner.com%3e Did we come up with another solution? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Logging command line at startup
On 9/15/09 3:18 PM, Paul Querna p...@querna.org wrote: I would prefer to just put it into the log file like Dan did.. not everyone uses apachectl :) +1 -- Brian Akins
Re: mod_cache, mod_deflate and Vary: User-Agent
On 8/26/09 3:20 PM, Paul Querna p...@querna.org wrote: I would write little lua scriptlets that map user agents to two buckets: supports gzip, doesnt support gzip. store the thing in mod_cache only twice, instead of once for every user agent. We do the same basic thing. We are moving towards the if you say you support gzip, then you get gzip attitude. I think less than 1% of our clients would be affected, and I think a lot of those are fake agents anyway. -- Brian Akins
Re: trunk - 2.2 proxy alignment
On 8/25/09 8:47 AM, Jim Jagielski j...@jagunet.com wrote: I'd like to float that idea again, before I spend any more time on it... +1 I've been, umm, struggling with some proxy stuff recently and it's annoying that 2.2.x and trunk are so different. One idea: Can we replace the void *context in the workers with a per module config? As is now, a single void * just seems dangerous. -- Brian Akins
Re: Apache HTTP Server development
Patches welcome. On 8/23/09 11:58 PM, tim robertson tim...@email.com wrote: . Events will help Apache server handle lots of concurrent connections because currently Apache cannot handle 10,000 concurrent connections on a low end single core server without Apache crashing or running very slow. FWIW, we very easily handle 50k+ concurrent connections on fairly modest hardware - 4 total cores, 4-8GB of RAM. (That's a very affordable server.) If you need to handle 10k concurrent clients and only have a single core box, you have other issues. -- Brian Akins
Balancer-manager, nonce, and xml
I found something in proxy and was wondering if it was intentional: In proxy_balancer, we check the nonce param and clear the params if incorrect or missing, however, then you can't view it in xml unless you include the nonce, because the xml param is no longer available. I was trying to get one of our monitoring tools to read the xml when I noticed it. I also am working on a simple thing to mark balancer members enable/disabled, but have to first scrap the HTML to get the value of nonce. This is rather a pain as well. Basically, I gave up on having a real HTTP health check mechanism within apache and was going to use the balancer manager stuff to enable/disable origins. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Segfault with fix for CVE-2009-1891
On 7/28/09 1:35 PM, Stefan Fritsch s...@sfritsch.de wrote: I have backported r791454 to 2.2.3 in Debian 4.0 and have received a report [1] about segfaults with mod_deflate and mod_php (5.2.0). Isn't php only officially supported via fastcgi? -- Brian Akins
Re: Hacking in an SLA for proxied requests in mod_proxy_http
I haven't looked at the code, but +1 for the idea. We had a hack that did something somewhat similar, but it was gross and in 2.0 - we never used it in prod. Is the proxy-timeout for the entire request to be returned, the first byte, or just an i/o timeout? -- Brian Akins
Re: Events, Destruction and Locking
This is how I envisioned the async stuff working. -Async event thread is used only for input/output of httpd to/from network* -After we read the headers, we pass the request/connection to the worker threads. Each request is sticky to a thread. Request stuff may block, etc, so this thread pool size is configurable and in mod_status, etc. -any writes out of the request to the clientare passed into the async thread. This may be wrapped in filters, whatever. *We may allow there to be multiple ones of these, ie one for proxies, or have a very well defined way to add watches to this. This is a very simplistic view. I was basically thinking that all conn_rec stuff is handled in the async event thread, all the request_rec stuff is handled in the worker threads. -- Brian Akins
Re: Events, Destruction and Locking
On 7/7/09 1:02 PM, Graham Leggett minf...@sharp.fm wrote: Ideally any async implementation should be 100% async end to end. I don't believe that its necessary though for a single request to be handled by more than one thread. True. However, what about things that may be process intensive. Ie, running lua in process. And we'd want to run multiple async threads (or processes). One of the issues with lighttpd with multiple processes (to use multiple cores, etc) is that lots of stuff is broken (ie, log files interleave). We just need to be aware of the issues that other servers have uncovered in this area. -- Brian Akins
Re: svn commit: r790205 - /httpd/httpd/trunk/modules/experimental/mod_noloris.c
On 7/2/09 8:37 AM, Nick Kew n...@webthing.com wrote: Not everyone who's concerned right now about slowloris has iptables at their disposal. Also, not everyone has access to the real IP very early in the connection phase. Some load balancers add the IP as a header. Generally speaking, most load balancers can handle many times more connections than the actual webservers (even when there are many webservers). Some load balancers per IP blocking schemes are either nonexistent or just don't work. Perhaps this is beyond the scope of this discussion, but it is an unfortunate reality for some folks. -- Brian Akins
Re: Mitigating the Slowloris DoS attack
On 6/22/09 10:40 PM, Weibin Yao nbubi...@gmail.com wrote: I have an idea to mitigate the problem: put the Nginx as a reverse proxy server in the front of apache. Or a device that effectively acts as such. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
A modest proposal, was Re: Mitigating the Slowloris DoS attack
On 6/23/09 12:48 AM, Paul Querna p...@querna.org wrote: Mitagation is the wrong approach. We all know our architecture is wrong. Another heretical suggestion: Lighttpd and nginx are both release under BSD-like licenses. Hear me out. I've actually been thinking how possible would it be to transform one of them into httpd 3.0? Nginx has a few architectural issues (a different cache for fasctcgi versus proxy??) and lighttpd is still fairly immature (cache can't handle Vary, lots of stuff broken when running multiple processes). However, just think if the forces of us and them combined (well, one of them). My personal pick is lighttpd - the community would fit better (nginx is almost all in Russian) and it already has a lot of Lua :) I know this would probably only even be considered in a bizzaro parallel universe. However, what are our alternatives? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: svn commit: r785425 - in /httpd/httpd/trunk: CHANGES modules/mappers/mod_dir.c
On 6/18/09 12:57 PM, Bob Ionescu bobsie...@googlemail.com wrote: DefaultMapping /index.php AddHandler php_handler .php Of course, though, php is only supported in fastcgi mode now. Unrelated to the discussion, but, yes, I will keep saying this over and over again until I quit seeing references to mod_php/libphp in apache2 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/10/09 8:52 AM, Rich Bowen rbo...@rcbowen.com wrote: Or something like that that's sufficiently like what we have now to be easy to learn, but sufficiently different to be harder to get wrong, and a whole heck of a lot easier to do dynamic vhosts. I think Lua (or whatever) could include enough user friendly glue to make something like that possible. If we had a standard set of variables (like, say, HOSTNAME or whatever) that could be interpolated into config directives, as well as a standard way to do variable assignment and backreferences, that would solve a whole class of problems that are really pretty difficult right now. That sounds a whole lot like using the config as a runtime. Hmmm, where have I heard that before ;) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/10/09 10:21 AM, Dan Poirier poir...@pobox.com wrote: That sounds a whole lot like using the config as a runtime. I'm not sure I follow that. I do like that the config would still look a lot like it does now, only more flexible. Even if we kept the same general syntax as now, if we introduce variables and if/else, then the config has to be compiled and ran at request time. (Glossing over a lot, I know.) Also, module would have to have some interface to accept the variables and conditionals. Also, we would have to basically write our own domain-specific language. My thought is rather than incrementally attempting to do this, we can just adopt an existing language that fits our needs - like lua. This adoption can be done incrementally, as well. Some examples of that, using a more complete version of mod_lua, have been mentioned. -- Brian Akins
Re: Some ramblings on httpd config
On 6/5/09 11:31 PM, Graham Dumpleton graham.dumple...@gmail.com wrote:
This last example wasn't even related to driving configuration. It was
in practice an actual handler hook implementation for request
processing, not configuration phases.
The way I see it, we have artificially separated configuration from request
processing. If you squint and tilt your head just right, you can see that
virtualhosts today are really just syntactical sugar over the if/else logic
inside of core:
Some pseudo request processing code to do same thing:
if listening_port == 80 then
if r.hostname == 'www.foo.com' then
elseif r.hostname =~ /www\d.bar.[org|net]/
end
end
Of course this could be further hidden from users with
macros/functions/libraries/modules...
Now, on the practical side, do we completely ditch the current config
system. Part of me says yes, but I know that will be -1'd to death. So,
I'd just like the ability to do something like this:
LoadModule lua_module mod_lua.so
Listen 80
LuaRequestHandler /path/to/my/lua/handler.lua
(or it can be inline Lua but have found that to be somewhat cumbersome)
Because I don't want to rewrite mod_proxy in lua, it'd be nice to have just
a little bit of glue that would allow me to use it in a more scripty sort
of way:
LoadModule proxy_module mod_proxy.so
LoadModule proxy_http_module mod_proxy_http.so
require httpd.proxy -- provided by mod_proxy glue
p = httpd.proxy.get_url('http://blah')
(Of course, that example could be handled like we do in mod_rewrite)
Currently, we can sorta do most request processing in lua. (FWIW, do the
request phases make any sense in a world where the entire request process is
handled by a script??) What is missing is the glue to the other, useful
parts of httpd - like cache, mod_dbd, proxy, etc.
Sure, one of us could hack together some example glue here and there, but
until we as a whole get why this is useful/important, it will be just
another list of patches waiting to be reviewed.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: A question about ap_content_length_filter()
On 6/9/09 7:50 AM, Peter Wang ptr.w...@gmail.com wrote: 8000 may be very specific on my server, i wonder where can i tuning such a limit. Yes 8000 is special in the apache response streaming. Are you using the deprecated mod_php or the supported fastcgi for php? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/3/09 7:50 PM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: 1. There are many and large and complex configurations out in the world. Which is exactly why I want/need a better way to do them. I'm currently using a template system to generate them. However I wind up with dozens (and dozens) of vhosts that sometimes only vary by a statement or too. A contrived example: -- lots of stuff if r.hostname == www.domain.com then -- a couple of things specific to this domain elseif r.hostname == www.domain2.com then ... Of course, this example may could have a little lua glue to handle this situation. Also, I'm not just talking about lua being the config language, I want lua to drive the httpd process. Ie, the above code gets ran on every request. 2. I admit that some improvements are needed. How about an approach that allows to embed a macro / scripting language into the current configuration system that allows you to do more advanced things if you need to. If we provided enough glue within our modules for lua, this this would be fairly easy. I already fake this a bit with mod_lua and handlers that do most of the work. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/4/09 7:51 AM, Dan Poirier poir...@pobox.com wrote:
and having to write a program to
implement mod_rewrite or a virtual host yourself.
Something like this would be trivial to do:
if prefix_match(r.uri, /foo/) then
return do_redirect(r, http://somewhere.else/;)
end
Okay, not a great example, but it's early here :) The fact that
conditionals are just *there* in lua would make some rewrite operations
easier. We could also provide plenty of helper stuff (written in lua,
probably).
rr = httpd.rewrite
rr.rule('/something(.*)', '/other$1')
-- in handler
x = rr.run( r )
if(x != DECLINED) then
return x
end
if r.env[something] then
return proxy:request(r, http://some.domain.com;)
end
Like I said, I'm not just thinking of lua as a config, but lua as the
driver of httpd. Or at least provide enough glue for some core modules to
allow us advanced users to do this. We may not need a whole lot of glue
to start with, a few useful ones:
serve_file(r, /some/file/on/disk)
x = do_subrequest(r, uri) -- x could contain headers, bucket brigade (or
just a big string)
do_redirect(r, location, 301)
Also, it would be nice if we could redo the mod_lua lua-request_rec glue so
that other modules can mixin addition functionality:
require httpd.util
r:serve_file(/some/file
r:redirect(location, 301)
I've been playing with that, but my lua-glue foo is weak. Something about
the way we are handling the metatable and __index doesn't like that.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/4/09 8:14 AM, Jorge Schrauwen jorge.schrau...@gmail.com wrote: Like Graham mentioned mod_macro can be of some use here. however since I'm looping in perl I may as well push the 4 lines needed to httpd instead of a one line macro replacemen. Okay, I'm not explaining my self well. I'm not just talking about generating the internal configuration from lua (or xml, or macro, etc.) But actually running httpd using lua. The request handling is done in lua - or rather driven by lua. See http://redmine.lighttpd.net/projects/lighttpd/wiki/AbsoLUAtion A side note: I think the vhost concept we have now is lacking. The separation that we have just isn't necessary and makes some common tasks hard. I'm not really willing to fight for this one though ;) I can fake what I want if I can just load up a lua handler to handle the request from post_read to handler. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Some ramblings on httpd config
Some ramblings from playing with other servers, particularly varnish and lighttpd. It would be interesting if we ditched the current configuration system and just used lua. I know this has been said before, but I've been doing a bit of playing with mod_lua and I think it it actually doable. We just need to supply a bit more glue. Some more suggestions based on this: -dump the current idea of virtual servers. This can be done programmatically in lua: if r.hostname == www.domain.com then r.document_root = /some/directory/ end We may could provide a lua library to somewhat simulate the current virtuals: require 'apache.httpd3.vhost' v = apache.httpd3.vhost:new() v.document_root = /blah v.servername = www.domain.com -We really only need three handlers: request handler (everything from post_read to handler), filter(s), and log. Once again some lua library could simulate different phases, if desired. -No need for rewrite, alias, etc. Can all be done in plain lua. I think lua makes more sense that trying to do logic in mod_rewrite, anyway. -config file is just a lua script: require 'apache.httpd3.server' require 'apache.httpd3.gzip' require 'apache.httpd3.cache' s = apache.httpd3.server:new() s.procs = 4 s.threadpool_size = 256 s.request_timeout = 30 s.default_error_log = /var/logs/error_log function output_filter( r ) apache.httpd3.gzip:do_gzip( r ) apache.httpd3.cache:do_cache_stuff( r ) end function logger( r ) -- log stuff end function handler( r ) -- do handler stuff set_output_handler(r, output_filter) set_logger(r, logger) end s.default_logger = logger s:add_handler(handler) s:main() This is possible with a very minimal httpd.conf and mod_lua today. You just use the httpd.conf to bootstrap the lua handler and register empty handlers for most everything. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/3/09 2:09 PM, Joachim Zobel jzo...@heute-morgen.de wrote: This does IMHO not address any of the problems users usually have and that are mainly due to a lack of validation. First of all, I don't really care about normal users, to be honest. Admit it, I'm not the only one. However, I do know that we can't just break everything for them. See http://people.apache.org/~rbowen/presentations/apacheconEU2005/hate_apache.pdf for what I consider a good description of the current problems. It solves most of the Missing page, I think. Also, if the lua doesn't compile, it's not a valid config. A few of the other points are addressed by using lua with some helper libraries. We could ease into this by having modules provide some lua glue for use in the lua handlers (proxy and cache in particular). -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
On 6/3/09 2:45 PM, Jorge Schrauwen jorge.schrau...@gmail.com wrote: I have to admit lua would be more flexible but I think most server admins have atleast come into contact with XML... while not necessarily the case with lua. XML with conditionals. Please, make it stop... I think we are going to have to choose one way (the current way, lua, something else) and just go with it. Nice thing about lua is that the configuration is the runtime script. There is no external and internal representation of things. (Well, not exactly, but close...). Lua is fast enough to just run every request - most of the hard stuff is still in C. Mod_rewrite goes away, which is a good thing :) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Some ramblings on httpd config
Has anyone looked at VCL that varnish uses? That and some playing with lighttpd's mod_magnet is what rekindled my interest: http://varnish.projects.linpro.no/wiki/Introduction http://varnish.projects.linpro.no/wiki/VCLExamples -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: How to Profile apache 2.2.11
On Linux, I've used oprofile to help with my module development. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Lua and memcache was Re: what is in modules vs what is in the core
OT, but will you contribute that to mod_lua ? :)
Based on mod_memcache and mod_wombat . For demonstration only as this uses
our hacked version, but you get the idea how simple this was. Build it in
lua dir like /usr/lib/lua/5.1/apach2/memcache.so
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
#include mod_memcache.h
#include mod_lua_request.h
typedef struct {
apr_memcache_t *memcache;
request_rec *request;
} lua_memcache_t;
static lua_memcache_t *check_memcache(lua_State * L, int index)
{
lua_memcache_t *lm;
luaL_checkudata(L, index, Apache2.Memcache);
lm = (lua_memcache_t *) lua_unboxpointer(L, index);
return lm;
}
static int get_client(lua_State * L)
{
lua_memcache_t *lm;
apr_memcache_t *m;
request_rec *r = lr_check_request_rec(L, 1);
m = ap_memcache_client(r-server);
lm = apr_pcalloc(r-pool, sizeof(lua_memcache_t));
lm-memcache = m;
lm-request = r;
lua_boxpointer(L, lm);
luaL_getmetatable(L, Apache2.Memcache);
lua_setmetatable(L, -2);
return 1;
}
/*these are fairly repetitive. can we combine this and not be slow?*/
/*only returns data for now and assumes it's a string*/
static int get_value(lua_State * L)
{
lua_memcache_t *lm = check_memcache(L, 1);
apr_memcache_t *m = lm-memcache;
request_rec *r = lm-request;
const char *key = luaL_checkstring(L, 2);
char *data = NULL;
apr_size_t len;
apr_status_t rv;
apr_uint16_t flags;
rv = apr_memcache_getp(m, r-pool, key, data, len, flags);
if(APR_SUCCESS != rv || NULL == data) {
lua_pushnil(L);
} else {
lua_pushstring(L, data);
}
return 1;
}
/*only returns data for now and assumes it's a string*/
static int set_value(lua_State * L)
{
lua_memcache_t *lm = check_memcache(L, 1);
apr_memcache_t *m = lm-memcache;
request_rec *r = lm-request;
const char *key = luaL_checkstring(L, 2);
char *data = (char *)luaL_checkstring(L, 3);
apr_uint32_t timeout = luaL_checkint(L, 4);
apr_size_t len;
apr_status_t rv;
apr_uint16_t flags = 0;
/*do we have to make a copy??*/
data = apr_pstrdup(r-pool, data);
len = strlen(data);
rv = apr_memcache_set(m, key, data, len, timeout, flags);
/*how to handle errors?*/
return 0;
}
static int incr_value(lua_State * L)
{
lua_memcache_t *lm = check_memcache(L, 1);
apr_memcache_t *m = lm-memcache;
request_rec *r = lm-request;
const char *key = luaL_checkstring(L, 2);
int n = 1;
int nv = 0;
int top = lua_gettop(L);
apr_status_t rv;
/*default to one, but if user gives another argument, use it*/
if(top 2) {
n = luaL_checknumber(L, 3);
}
rv = apr_memcache_incr(m, key, n, nv);
/*XXX: need to deal with errors. what if rv != APR_SUCCESS*/
lua_pushnumber(L, nv);
return 1;
}
static int decr_value(lua_State * L)
{
lua_memcache_t *lm = check_memcache(L, 1);
apr_memcache_t *m = lm-memcache;
request_rec *r = lm-request;
const char *key = luaL_checkstring(L, 2);
int n = 1;
int nv = 0;
int top = lua_gettop(L);
apr_status_t rv;
/*default to one, but if user gives another argument, use it*/
if(top 2) {
n = luaL_checknumber(L, 3);
}
rv = apr_memcache_decr(m, key, n, nv);
/*XXX: need to deal with errors. what if rv != APR_SUCCESS*/
lua_pushnumber(L, nv);
return 1;
}
static const struct luaL_Reg memcache_funcs[] = {
{client, get_client},
{NULL, NULL}
};
static const struct luaL_Reg memcache_methods[] = {
{get, get_value},
{set, set_value},
{incr, incr_value},
{decr, decr_value},
{NULL, NULL}
};
int luaopen_apache2_memcache(lua_State * L)
{
luaL_newmetatable(L, Apache2.Memcache);
lua_pushvalue(L, -1);
lua_setfield(L, -2, __index);
luaL_register(L, NULL, memcache_methods);
luaL_register(L, memcache, memcache_funcs);
return 1;
}
Re: Lua and memcache was Re: what is in modules vs what is in the core
On 3/31/09 10:18 AM, Bertrand Mansion bmans...@mamasam.net wrote: Reading the comments in your code, it seems that mod_lua could benefit from a better error reporting ? Perhaps. Or I just needed to actually add error checking Right now I rely on the lua side to sanity check stuff. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Profiling Apache
On 1/28/09 1:25 AM, Paras Fadte plf...@gmail.com wrote: Hi, Which tool is the most suitable for profiling Apache ? gprof ? Oprofile ? I've used oprofile. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: accept mod_fcgid codebase into httpd project
On 1/11/09 10:53 PM, William A. Rowe, Jr. wr...@rowe-clan.net wrote: [ ] Accept mod_fcgid into httpd +1 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Mod_memcache status
Anyone know anything about status of this? http://code.google.com/p/modmemcache/ -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ResponseHeaders to Environment
Seems like your own module would be small and straight forward. There is a patch floating around for mod_setenvif that adds a filter for it to act on response headers. That may could be used, depending on the order things happen. I'd still like this patch to be committed, I can resubmit, if you want to take a look. Of course, this would be easy to do in lua... ;) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Memory leak in mod_proxy_http and in mpm_event components.
On 10/30/08 12:48 PM, Manik Taneja [EMAIL PROTECTED] wrote: So it seems that there is a slow-leak perhaps we might be hitting the problem that has been described in the following post. http://www.nabble.com/apr-pools---memory-leaks-td19766166.html FWIW, we always run with MaxRequestsPerChild set to something. Usually a fairly large number, but they recycle a few times a day or so. Start enough children, and we don't really notice a performance hit when the children are recycled. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: MPMs, COW vs Child Process Spawning
One of the things I've noticed on a practical level, is that forked children in worker wind up being linearized within themselves to some extent. Think of how the buffered logs work in mod_log_config (one buffer protected by a mutex). (I can't think of the correct term for it right now. Basically every thread is waiting on the same mutex eventually.) This is seen in other modules... The fork gets around this somewhat, as there are multiple mutexes independently locked/unlocked. If we have one multi-threaded process, this will force us to think about the performance ramifications of these uses of single mutexes. For mod_log_config, for example, a queue with a single writer thread may be the answer. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: MPMs, COW vs Child Process Spawning
On 10/29/08 3:40 PM, Paul Querna [EMAIL PROTECTED] wrote: Of course, maybe that is a good thing, since then they would be ported forward and work on win32 too. Can you delete open files in win32 yet?? If not, most of my stuff will not port... -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: strange usage pattern for child processes
On 10/18/08 4:28 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Also consider today that a server on broadband can easily spew 1gb/sec bandwidth at the client. If this is composed content (or proxied, etc, but not sendfiled) it would make sense to allow multiple buffer pages and/or resizing buffer pages, in a Location or Files or Proxy specific context. Would not the generic store-and-forward approach I sent last week help all of these situations? It effective turns any request into a sendfiled response. Let me do some checking and I may be able to just donate the code, since it's basically a very hacked up mod_deflate crossed with mod_xsendfile. It works for us, but we don't use the stock proxy, so not 100% it will help if the frontend and backend pools are somehow linked. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Environment confusion
On 10/20/08 1:19 PM, Graham Leggett [EMAIL PROTECTED] wrote: This will make some interesting templating options possible, and will probably make lives easier for people doing mass hosting. Seems like a place to get a lot of bug reports as well. I choose to just use a real template system to handle configs. I'm sure I'm not the only one. I'm +0 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: strange usage pattern for child processes
On 10/15/08 6:56 PM, Graham Leggett [EMAIL PROTECTED] wrote:
Obviously, if the loop comes round more than once, then the client comes
into play. This definitely needs to be fixed, it is a big performance issue.
Could a more general purpose optimization be done? I was thinking of a
generic store and forward filter. Basically, just dump the entire brigade
into some storage (memory, file) in a loop (like deflate sorta does) without
sending to client at all until the response is finished. This would work
for proxy, php, some strange handler, etc.
A small test I did was a module had a filter that just dumped all the
brigade contents into a temp file. When it saw EOS, it would then send the
complete file to the client (via sendfile, mmap, etc..) and remove the
tempfile. Almost like an in process X-Sendfile. This very rudimentary
module increased overall throughput of some large SSI files by about 20% and
completely shielded origin servers from clients in some proxy tests. It
also did not consume very much memory (although I was writing the temp files
into /dev/shm on Linux).
Basic logic - I ignored flush and meta buckets in my tests:
while (!APR_BRIGADE_EMPTY(bb)) {
e = APR_BRIGADE_FIRST(bb);
if (APR_BUCKET_IS_EOS(e)) {
create a file bucket with the tempfile and send it along
tempfile was opened with APR_DELONCLOSE
} else {
apr_bucket_read(e,
write content to temp file
}
apr_bucket_delete(e);
}
apr_brigade_cleanup(bb);
return APR_SUCCESS;
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: Apache httpd 2.2.10 test tarballs available...
On 10/7/08 8:49 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Sure sounds like this is a re-initialization of mysql, with apr and php fighting for the honors. I thought the official support of php was fastcgi only in httpd 2.2 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
PHP was Re: Apache httpd 2.2.10 test tarballs available...
On 10/8/08 2:15 PM, Oden Eriksson [EMAIL PROTECTED] wrote: vhosting does not work very well with apache, without band-aids like fastcgi etc. that's a shame. There is a rather lengthy list of reasons why mod_php doesn't work correctly. AFAIK, the official word has been to use fast-cgi for almost 3 years. Most other major http servers have taken the same stance. I think we (httpd folks in general) have not done a good job of documenting this. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: load balancer workers
On 10/8/08 4:06 PM, Paul Querna [EMAIL PROTECTED] wrote: I don't think the approach you are suggesting will work. I would suggest duplicating the workers at startup, enable/disable in each balancer to 'move' them. +1. Do graceful restarts work correctly with the stock load balancer? If so, they are usually very cheap and non disruptive if you need this functionality. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: PHP was Re: Apache httpd 2.2.10 test tarballs available...
This is getting of topic in dev, but... On 10/8/08 4:17 PM, Arturo 'Buanzo' Busleiman [EMAIL PROTECTED] wrote: I've migrated all of my webservers to a mixture of Lighttpd and Nginx, Why? The apache is bloated and slow argument is just plain incorrect. (FWIW, I have nothing against eitehr of those other than the FUD they spread about apache.) BUT there are some PHP apps that do not work ok with fcgi. So, they wouldn't work on lighttpd, I suppose. For them I use Apache. I use a lot of Apache for reverse-proxy setups, for instance, using lighttpd to serve static content, but keep Apache in the front for mod_security, for example. Mod_security can front fastcgi quite easily. There may be some instances where it doesn't work for all types of rules. (?) The point I was making is that php is not supported via mod_php in httpd 2.x. Can you get it to work, yes, but if you ask question in php-lanf they will tell you use apache 1.3 or fastcgi. Same in httpd-land. Since this fact has not been well publicized, people keep using mod_php and php developers never actually test their code under fact-cgi. Should someone draft an official statement that can be put on httpd.apache.org? It would help if we included an implementation of fact-cgi, as well. Or, at least, gave some links to the more popular one(s). -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [PATCH] add ap_sendfile_enabled
On 9/30/08 1:15 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: should we look at something closer to ap_mpm_query (ap_core_query?) and start adding it a whole lot of stuff? +1 Something to ponder; injecting this as-is for 2.2.10 seems rushed. +1 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [PATCH] add ap_sendfile_enabled
On 9/25/08 9:37 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Couple of points, API fn name sucks, can it be ap_request_config_test(r,feat)? Sure. Always request-rec based, there should be no need for server rec (two args are faster than 3). True. About your other idea, it's a no-go. For the record, it was Paul's idea to add a const char* arg... We need to do something or users - including ourselves - will do the very bad thing of declaring CORE_PRIVATE all over the place. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
[PATCH] add ap_sendfile_enabled
Got tired of having to set core_private for one bit of info. How does this
look?
Index: server/request.c
===
--- server/request.c(revision 699033)
+++ server/request.c(working copy)
@@ -1907,3 +1907,14 @@
return (r-main == NULL) /* otherwise, this is a sub-request */
(r-prev == NULL); /* otherwise, this is an internal
redirect */
}
+
+AP_DECLARE(int) ap_sendfile_enabled(request_rec *r)
+{
+int rc = 0;
+#if APR_HAS_SENDFILE
+core_dir_config *conf = ap_get_module_config(r-per_dir_config,
+ core_module);
+rc = conf-enable_sendfile == ENABLE_SENDFILE_OFF;
+#endif
+return rc;
+}
Index: include/httpd.h
===
--- include/httpd.h(revision 699033)
+++ include/httpd.h(working copy)
@@ -1541,6 +1541,13 @@
AP_DECLARE(int) ap_count_dirs(const char *path);
/**
+ * Is sendfile enabled for this request
+ * @param r the current request
+ * @return 1 if enabled, 0 is not
+ */
+AP_DECLARE(int) ap_sendfile_enabled(request_rec *r);
+
+/**
* Copy at most @a n leading directories of @a s into @a d. @a d
* should be at least as large as @a s plus 1 extra byte
*
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: Future direction of MPMs, was Re: svn commit: r697357 - in /httpd/httpd/trunk: include/ modules/http/ modules/test/ server/ server/mpm/experimental/event/
On 9/21/08 2:17 AM, Bing Swen [EMAIL PROTECTED] wrote: But an optimal network i/o model needs a layer that maps a *request* to a thread, so that a worker thread (or process) will not have to be tied up entirely with a single connection during the whole life time of the connection. Instead, a worker can be scheduled to handle different connections, which helps both reducing the number of workers and the performance of request handling (especially on slow connections). I still want to see this backed up with real world experience. I know I keep repeating myself, but in the real world, we have never seen the supposed inherent performance problems in the worker model (1 connection = 1 thread). It sounds great to theorize about the wonders of a completely event driven or asynchronous model. However, it seems that this only nets real measurable performance gains is very simplistic benchmarks. I'm all for making httpd faster, scale better, etc. I just don't want to be extremely disappointed if we rewrite it all and gain nothing but a more complicated model. If we get great gains, wonderful, but I'd like to see some actually numbers before we all decided to rework the core. (Disclaimer: yes, I'm partially playing devil's advocate here.) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: mod_proxy + mod_mem_cache memory consumption issues
Have you tried mod_disk_cache with the cachroot in /dev/shm? Mod_mem_cache, it seems, is negleted in comparison to the disk one. Using /dev/shm is basically putting the cache in shared emmory. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [community] 2.3.0 alpha on October 1?
On 9/4/08 4:22 AM, Bing Swen [EMAIL PROTECTED] wrote: Maybe you have super-optimized hardware and system? Nope, standard off the shelf systems. So I wonder perhaps next time you guys may bother to take time to also run Nginx on your platform and tell us how it performs against your httpd build... Will do. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [community] 2.3.0 alpha on October 1?
On 9/4/08 2:54 AM, steve [EMAIL PROTECTED] wrote: Filling gigE and requests/sec aren't targets that concern me as a user. They sound like nice beanchmarks though... To some folks, filling a gigE is important. FWIW, I ran some tests yesterday with about 50K ISDN (384K) speed clients and httpd and the system was still very responsive. Will try to do some ~ 56k tests if I have time today. (Speaking of pet peeves -- why does Apache handle so many things besides HTTP, and yet I have to get other servers to handle certain kinds of HTTP requests because Apache doesn't handle it well?) Any particular requests you'd like to address here? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [community] 2.3.0 alpha on October 1?
On 9/1/08 8:11 AM, Jim Jagielski [EMAIL PROTECTED] wrote: On Aug 31, 2008, at 9:49 AM, Bing Swen wrote: To my knowledge, the one thread per connection network i/o model is a suboptimal use threads vs. events is certainly not, imo, a finalized debate yet with a known winner or loser. Maybe 5-10 years ago events had a clear advantage but today that is hardly the case... I have documented my vote on this in the past, but the async is inherently faster looks good on paper, but does not, IMNSHO, measure up in the real world. It seems that lighttpd, and others, simply come with a much more realistic default configuration out of the box. Our latest builds, on some fairly modest hardware, are actually about twice as fast as numbers I posted last time async-vs-threads came up. Apache can very easily fill multiple gigE interfaces on modest hardware. We can sustain about 45k requests/sec on our build on a dual dual-core system with a network card that supports Linux NAPI (that made a huge difference). Without much tuning 35k is pretty easy. (Note: this was very small files, bcs it's so easy to fill the network interfaces). Apache is slow is just FUD, plain and simple. We need to work on getting things like x-sendfile into stock distribution and pushing the use fastcgi for php type things into the documentation. (Sorry, put this area is one of my pet peeves.) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [community] 2.3.0 alpha on October 1?
On 9/2/08 1:02 PM, bing swen [EMAIL PROTECTED] wrote: There is a little different viewpoint. According to some recent test reports comparing Apache 2.2 and Nginx 0.6/0.7 (from a blog website admin.), Apache could do as well as Nginx when there are a few connections each of which carries many many requests. The hard time for Apache came when there are many *slow connections*, each of which sends only a few small requests at large durations. In such condition, Apache could hardly respond when the no. of connections reached over 3,000 (even when there is still much free memory). I saw this comparison somewhere. It just does not seem to match what I have seen. Our little ole website has been known to take a few connections from slow clients, but we have not really seen this slow down. I'd like to see more specifics of the test. All of my numbers are from real world stuff. (FYI, NAPI is just a way which Linux handles NIC irq's better - a gross simplification, but it makes a huge difference on very busy web servers. I'm sure other OS's have something similar?) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [community] 2.3.0 alpha on October 1?
On 9/2/08 3:15 PM, bing swen [EMAIL PROTECTED] wrote: It seems the test (done by another guy) indeed used an everything plus the kitchen sink default Apache httpd at first, but then dropping off 3/4 of all of the default modules (maybe not that much, but only for serving static pages) seemed to have a not big help. The MPM used the default configuration (prefork on Linux and threads on Windows). [not your settings?] Egads, no wonder they got such horrible performance. Worker (or event, maybe) seems to be the best way to go, at least based on our testing. People who do performance benchmarks on Linux with default config and prefork are careless at best.We run a fairly full featured config, but try to be smart about it. We run internal benchmarks often and collect a lot of performance statistics from live traffic (response times, etc.) We also, however, do not run any vendor supplied httpd build, either. RedHat's, for example, is not for production websites, IMO. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: mod_proxy_balancer enhancements
On 8/28/08 8:45 PM, Jess Holle [EMAIL PROTECTED] wrote: Possible solutions include: Having an option to have a background thread ping the backends rather than allowing normal requests to do so. One way to possibly do this is to have a single health checking that's in the parent that can notify children of origin server changes. Could just be setting some flags in shared memory. The health-checker could be an external process managed like a logger as well or just forked from parent. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Mpm maintenance thread hook
On 8/4/08 3:41 PM, Jim Jagielski [EMAIL PROTECTED] wrote: *grin* I've always wondered what usefulness ap_run_monitor(p) FWIW, we use the monitor hook all the time... Most of our stuff uses some shared memory and the monitor hook in parent, but I have considered hacking something into each child (I may have actually done that before...). We rarely have overly idle children, so we do have something that does like what Paul was suggesting. We used one of the really late connection hooks (I think, I'm away from my normal computing spot), so we didn't hack the MPM, but what Paul suggests sound plausible. Of course, we run more threads per child than a lot of people, so an extra utility thread would not be such a big deal and may keep some stuff cleaner. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Mpm maintenance thread hook
On 8/4/08 12:59 PM, Mladen Turk [EMAIL PROTECTED] wrote: mod_balancer decaying (averaging) the transferred statistics, so that node-up/recovery doesn't consume all the load. mod_proxy_ajp sending ping/pong at regular intervals (detecting node failure before request) mod_cache probably lots of use cases since its time interval related Couldn't these be in the monitor hook in parent, assuming some shared memory tied the children together? I'm +1 on the idea. For 99% of my stuff, the servers get access frequently enough that it doesn't matter, but I have had to hack together a few child maintenance threads. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Kill or Stop Http Request Processing
On 7/23/08 2:51 PM, rhiren [EMAIL PROTECTED] wrote: How can I stop the processing of that request on the server, so that it does not spend any more time on the document content processing and frees up the CPU and memory. Check r-aborted -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: mod_disk_cache jumbopatch - 2.2.9 version
On 6/19/08 5:35 PM, Niklas Edmundsson [EMAIL PROTECTED] wrote: Brian: I remember you talking about some in-house modifications using DBM or something to track accesses to cached data and using it to find candidates to remove. Care to share? On every cache store, we write out to a pipe some info (url, filename, time, expire time, size, and probably some more stuff). This pipe is read by a really simple perl script that sticks this stuff in a mysql database. The actual cache manager is a fast-cgi script for manual ejections and is ran on the command line via cron to prune the cache. (it includes a small bit on Inline C.) This was done as a prototype, but works so well that we kept it. I think this was in one of my apachecon presentations. I'll try to look it up. Wish I could share source... That's a discussion I have been having here for 5 years... BTW, XFS sucked for us. It would randomly go into la-la land and nothing could be read or written. A mildly tuned ext3 seems to be good enough, although some of our caches live in /dev/shm... -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Mod_cache providers question
Okay, started looking at mod_cache and had a question about the way it is handling providers. We call ap_cache_get_providers which seems to do a lot of work that we could probably do at config time rather than run time. Any reason why providers can't be attached to uri's at config time and avoid a lot of this? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: 2.2.9 status
On 6/2/08 2:02 PM, Jim Jagielski [EMAIL PROTECTED] wrote: Please review STATUS and prepare for tarball testing in a coupla days! FWIW, if my vote counts: *htpasswd: Fix salt generation weakness. PR 31440 +1 * mod_unique_id: Convert request time to seconds before before storing it in +1 -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Adding purge/invalidation to mod_cache
How we handle purge: -Our cache is only disk based, so it is somewhat easier. -We have a hook that gets called after each cache store - ie, after we write a vary meta file, after we write a real meta file, after a datafile write (on close). -We also have a cache_get_info function that is exported (think curl_get_info) so that other modules can get info about cache objects (filenames, uri, sizes, etc.) -Our default cache store registers this post-store hook. It just writes the data out to a pipe - uri filename size expires\n -This piped program streams this info into a database (MySQL usually, as we had issues with SQLite) -Our cache manager is really just a fastcgi that uses the database to expire and purge cache objects. In a vary situation, we will get 3 post-store's for the initial cache: -first is the main key for main meta file -second is the varied key for the real meta file (the one that has headers, etc.) -third is the varied key/file for the data So a simple sql query for select file from my_table where url='/my/url' will return all 3. (LIKE and regex's work as well.) We also can easily keep track of cache size (select sum(object_size) from my_table), oldest entries, etc. Subsequent requests for same uri with different vary will add 2 entries to the database - varied meta and data. The fastcgi and pipe are because we prototyped all of this in perl and never rewrote it because it all works fairly well. Also, we had a few issues with mysql client directly in Apache and the pipe was just easier... -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: On future mod_wombat improvements
FWIW, we already use apreq for other stuff anyway. Also, it does a lot of useful stuff that can be exposed via mod_wombat One of my top things I want is a way to generically register and call some lua code that takes advantage of resource pools, caching, etc. No offense to Brian M, but I found it easier to start from scratch rather than hack on mod_wombat. Granted, I used a ton of ideas from mod_wombat, but it would have taken more time to fix the implementation. There is an early version of my efforts floating around - mod_lua_request. (I can repost it, as well). With the register/call interface, I do all the hook stuff (fixups, handler, etc.) in a separate module. The base module is just for the apache-lua glue. I still hack some on mod_wombat, to be fair... -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
