Re: Possible new cache architecture
On Wed, May 03, 2006 at 02:07:44PM +0200, Plüm, Rüdiger, VF EITO wrote: > > -Ursprüngliche Nachricht- > > Von: Joe Orton > > > > The way I would expect it to work would be by passing f->next in to > > the store_body callback, it looks doomed to eat RAM as currently > > designed. mod_disk_cache's store_body implementation can then do: > > > > 1. read bucket(s) from brigade, appending to some temp brigade > > 2. write bucket(s) in temp brigade to cache file > > 3. pass temp brigade on to f->next > > 4. clear temp brigade to ensure memory is released > > 5. goto 1 > > Yes, this was also my idea, but I would like to avoid this, because: > > 1. This is an API change which might be hard to backport. > 2. I do not really like the close tie between the storage provider >and the filter chain. It forces the provider to do things it >should not care about from my point of view. At least this much could be solved I suppose by passing in a callback of type apr_brigade_flush which does the pass to f->next; the storage provider could remain filter-agnostic then. No idea about your other issues, sorry. joe
Re: apr_brigade_insert_file() LFS/Linux issues
On Wed, May 03, 2006 at 05:46:09PM +0200, Niklas Edmundsson wrote:
> On Wed, 3 May 2006, Joe Orton wrote:
>
> >>I've run into apr_brigade_insert_file() creating brigades that's not
> >>possible to sendfile() (EINVAL), this is with httpd-2.2.2 on Ubuntu
> >>Breezy Linux amd64 (64bit). The file in question is 4.3GB, and it
> >>seems that sendfile() doesn't cope with that.
> >>
> >>Has anyone else seen this?
> >
> >No, works fine here. Can you get strace output for the failing call?
>
> [pid 931] open("/httpcache/HU/zG/8j/_HJBmSBIt0F2CnvQ", O_RDONLY) = 11
> ...
> [pid 931] sendfile(10, 11, [4096], 4571090944) = -1 EINVAL (Invalid
> argument)
OK, what filesystem? Colm had reported the same thing on Debian/IA64,
which was on an NFS mount, IIRC. If you adjust apr_brigade_insert_file
to only allow buckets of MAX_BUCKET_SIZE regardless, i.e. change the
conditional to:
if (length < MAX_BUCKET_SIZE) {
does that work?
joe
[STATUS] (httpd-2.1) Wed May 3 23:53:24 2006
APACHE 2.3 STATUS: -*-text-*-
Last modified at [$Date: 2006-02-02 16:28:52 -0500 (Thu, 02 Feb 2006) $]
The current version of this file can be found at:
* http://svn.apache.org/repos/asf/httpd/httpd/trunk/STATUS
Documentation status is maintained seperately and can be found at:
* docs/STATUS in this source tree, or
* http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/STATUS
Consult the following STATUS files for information on related projects:
* http://svn.apache.org/repos/asf/apr/apr/trunk/STATUS
* http://svn.apache.org/repos/asf/apr/apr-util/trunk/STATUS
Patches considered for backport are noted in their branches' STATUS:
* http://svn.apache.org/repos/asf/httpd/httpd/branches/1.3.x/STATUS
* http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/STATUS
* http://svn.apache.org/repos/asf/httpd/httpd/branches/2.2.x/STATUS
Release history:
[NOTE that x.{odd}.z versions are strictly Alpha/Beta releases,
while x.{even}.z versions are Stable/GA releases.]
2.3.0 : in development
Contributors looking for a mission:
* Just do an egrep on "TODO" or "XXX" in the source.
* Review the bug database at: http://issues.apache.org/bugzilla/
* Review the "PatchAvailable" bugs in the bug database:
https://issues.apache.org/bugzilla/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&product=Apache+httpd-2&keywords=PatchAvailable
After testing, you can append a comment saying "Reviewed and tested".
* Open bugs in the bug database.
CURRENT RELEASE NOTES:
RELEASE SHOWSTOPPERS:
* Handling of non-trailing / config by non-default handler is broken
http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=105451701628081&w=2
jerenkrantz asks: Why should this block a release?
wsanchez agrees: this may be a change in behavior, but isn't
clearly wrong, and even if so, it doesn't seem like a
showstopper.
* the edge connection filter cannot be removed
http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=105366252619530&w=2
jerenkrantz asks: Why should this block a release?
stas replies: because it requires a rewrite of the filters stack
implementation (you have suggested that) and once 2.2 is
released you can't do that anymore.
CURRENT VOTES:
* If the parent process dies, should the remaining child processes
"gracefully" self-terminate. Or maybe we should make it a runtime
option, or have a concept of 2 parent processes (one being a
"hot spare").
See: Message-ID: <[EMAIL PROTECTED]>
Self-destruct: Ken, Martin, Lars
Not self-destruct: BrianP, Ian, Cliff, BillS
Make it runtime configurable: Aaron, jim, Justin, wrowe, rederpj, nd
/* The below was a concept on *how* to handle the problem */
Have 2 parents: +1: jim
-1: Justin, wrowe, rederpj, nd
+0: Lars, Martin (while standing by, could it do
something useful?)
* Make the worker MPM the default MPM for threaded Unix boxes.
+1: Justin, Ian, Cliff, BillS, striker, wrowe, nd
+0: BrianP, Aaron (mutex contention is looking better with the
latest code, let's continue tuning and testing), rederpj, jim
-0: Lars
pquerna: Do we want to change this for 2.2?
RELEASE NON-SHOWSTOPPERS BUT WOULD BE REAL NICE TO WRAP THESE UP:
* Patches submitted to the bug database:
http://issues.apache.org/bugzilla/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&product=Apache+httpd-2.0&keywords=PatchAvailable
* Filter stacks and subrequests, redirects and fast redirects.
There's at least one PR that suffers from the current unclean behaviour
(which lets the server send garbage): PR 17629
nd says: Every subrequest should get its own filter stack with the
subreq_core filter as bottom-most. That filter does two things:
- swallow EOS buckets
- redirect the data stream to the upper request's (rr->main)
filter chain directly after the subrequest's starting
point.
Once we have a clean solution, we can try to optimize
it, so that the server won't be slow down too much.
* RFC 2616 violations.
Closed PRs: 15857.
Open PRs: 15852, 15859, 15861, 15864, 15865, 15866, 15868, 15869,
15870, 16120, 16125, 16126, 16133, 16135, 16136, 16137,
16138, 16139, 16140, 16142, 16518, 16520, 16521,
jerenkrantz says: need to decide how many we need to backport and/or
if these rise to showstopper status.
wrowe suggests: it would be nice to see "MUST" v.s. "SHOULD" v.s. "MAY"
out of this list, without reviewing them individually.
* There is a bug
[STATUS] (httpd-2.0) Wed May 3 23:51:35 2006
APACHE 2.0 STATUS: -*-text-*- Last modified at [$Date: 2006-05-01 08:57:25 -0400 (Mon, 01 May 2006) $] The current version of this file can be found at: * http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/STATUS Documentation status is maintained seperately and can be found at: * docs/STATUS in this source tree, or * http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/docs/STATUS Consult the following STATUS files for information on related projects: * http://svn.apache.org/repos/asf/apr/apr/branches/0.9.x/STATUS * http://svn.apache.org/repos/asf/apr/apr-util/branches/0.9.x/STATUS Consult the trunk/ for all new development and documentation efforts: * http://svn.apache.org/repos/asf/httpd/httpd/trunk/STATUS * http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/STATUS Release history: 2.0.59 : In development 2.0.58 : released May 1, 2006 as GA. 2.0.57 : Tagged on April 19, 2006. Not released. 2.0.56 : Tagged on April 16, 2006. Not released. 2.0.55 : released October 16, 2005 as GA. 2.0.54 : released April 17, 2005 as GA. 2.0.53 : released February 7, 2005 as GA. 2.0.52 : released September 28, 2004 as GA. 2.0.51 : released September 15, 2004 as GA. 2.0.50 : released June 30, 2004 as GA. 2.0.49 : released March 19, 2004 as GA. 2.0.48 : released October 29, 2003 as GA. 2.0.47 : released July 09, 2003 as GA. 2.0.46 : released May 28, 2003 as GA. 2.0.45 : released April 1, 2003 as GA. 2.0.44 : released January 20, 2003 as GA. 2.0.43 : released October 3, 2002 as GA. 2.0.42 : released September 24, 2002 as GA. 2.0.41 : rolled September 16, 2002. not released. 2.0.40 : released August 9, 2002 as GA. 2.0.39 : released June 17, 2002 as GA. 2.0.38 : rolled June 16, 2002. not released. 2.0.37 : rolled June 11, 2002. not released. 2.0.36 : released May 6, 2002 as GA. 2.0.35 : released April 5, 2002 as GA. 2.0.34 : tagged March 26, 2002. 2.0.33 : tagged March 6, 2002. not released. 2.0.32 : released Feburary 16, 2002 as beta. 2.0.31 : rolled Feburary 1, 2002. not released. 2.0.30 : tagged January 8, 2002. not rolled. 2.0.29 : tagged November 27, 2001. not rolled. 2.0.28 : released November 13, 2001 as beta. 2.0.27 : rolled November 6, 2001 2.0.26 : tagged October 16, 2001. not rolled. 2.0.25 : rolled August 29, 2001 2.0.24 : rolled August 18, 2001 2.0.23 : rolled August 9, 2001 2.0.22 : rolled July 29, 2001 2.0.21 : rolled July 20, 2001 2.0.20 : rolled July 8, 2001 2.0.19 : rolled June 27, 2001 2.0.18 : rolled May 18, 2001 2.0.17 : rolled April 17, 2001 2.0.16 : rolled April 4, 2001 2.0.15 : rolled March 21, 2001 2.0.14 : rolled March 7, 2001 2.0a9 : released December 12, 2000 2.0a8 : released November 20, 2000 2.0a7 : released October 8, 2000 2.0a6 : released August 18, 2000 2.0a5 : released August 4, 2000 2.0a4 : released June 7, 2000 2.0a3 : released April 28, 2000 2.0a2 : released March 31, 2000 2.0a1 : released March 10, 2000 Contributors looking for a mission: * Just do an egrep on "TODO" or "XXX" in the source. * Review the bug database at: http://issues.apache.org/bugzilla/ * Review the "PatchAvailable" bugs in the bug database: http://issues.apache.org/bugzilla/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&product=Apache+httpd-2.0&keywords=PatchAvailable After testing, you can append a comment saying "Reviewed and tested". * Open bugs in the bug database. CURRENT RELEASE NOTES: * Forward binary compatibility is expected of Apache 2.0.x releases, such that no MMN major number changes will occur. Such changes can only be made in the trunk. * All commits to branches/2.0.x must be reflected in SVN trunk, as well, if they apply. Logical progression is commit to trunk, get feedback and votes on list or in STATUS, then merge into branches/2.2.x, and finally merge into branches/2.0.x, as applicable. RELEASE SHOWSTOPPERS: PATCHES ACCEPTED TO BACKPORT FROM TRUNK: [ start all new proposals below, under PATCHES PROPOSED. ] PATCHES PROPOSED TO BACKPORT FROM TRUNK: [ please place SVN revisions from trunk here, so it is easy to identify exactly what the proposed changes are! Add all new proposals to the end of this list. ] *) Reverse Proxy fixes: bug and Cookie support Patch is at http://people.apache.org/~colm/httpd-2.0-reverse-proxy-cookie.patch and is in production with Clients. +1: niq +1: wrowe; looks good, no way to apply this without a minor bump *) Backport 102870; PR 17217; stop linking OpenSSL to support/* binaries (especially when compiled --with-static-support (!))
Re: Generic cache architecture
Ruediger Pluem wrote: On 05/03/2006 11:27 PM, William A. Rowe, Jr. wrote: Moreso, we need more third party authors to -participate- in telling us what in HTTPD-2.4 will make their module better. And a faster cycle of 6mos-1yr gives them a chance to do this and realize the benefits in the official release more quickly. But some of them will fall off the shelf (I guess especially some commercial ones), because they do not want to invest time again into an changing API. Of course. Why stall all efforts because some don't move with new technologies? This means that they stick with an older version and do not do releases for each major version, but lets say for every second. Ok, their user's loss, not an httpd development issue. Let's be clear here, this isn't a statement against making their lives easier to actually do the ports - we could be a heck of a lot more helpful in module authoring and especially in porting documetation. Furthermore having frequent major releases increases the backport requests from user side and thus likely the backport efforts, as some users stick with older versions for whatever reasons (e.g. third party modules). Wrong. We honor fewer backport requests for features. We might entertain some for bug fixes certainly. But the answer's always upgrade asap for fixes, and an upgrade's manditory for new features. Backporting features is a vicious cycle. No problem here with folks using Apache 1.3 when it solves their pain. But if they want cool feature X and we give it to them, we support 1.3 for that much longer because this-bug and that-bug aught to get fixed, and new feature X has a bug so we have a subsequent release, followed by 12 more pleas for other features to be backported. Nip it in the bud, freeze the features in old version, and poof, people move because they *want* the new features, it solves more of their pain, and so they have an incentive to make *their* investment of time in migrating. Take away the incentive and they will not (heck, should not) migrate up to our supported version. Maybe we should have a FDT (Frequently Discussed Topics) to collect the arguments ;-). Hehe.
Re: Generic cache architecture
On Wednesday 03 May 2006 20:44, Brian Akins wrote: > Is anyone else interested in having a generic cache architecture? (not > http). I have plenty of cases were I re-invent the wheel for caching > various things (IP's, sessions, whatever, etc.). It would be nice to > have a provider based architecture for such things. Yes, I think at that basic level, your proposal is uncontroversial. I'd like to be able to plug in alternative cacheing modules without having to reimplement the whole thing. I would point out there's a big "grey area" here, with regimes such as ESI cacheing that are bastardised HTTP. mod_cache_http is (modulo any bugs) technically accurate for the current cache module, but calling it mod_cache_rfc2616 might be less confusing for users of other cacheing regimes that purport to be HTTP (like ESI), or that run *on top of* HTTP (like some XML-based monstrosities). -- Nick Kew
Re: Generic cache architecture
On 05/04/2006 12:35 AM, Justin Erenkrantz wrote: > > For simplicity sake, I agree. Let's call this new thing > mod_cache_generic or mod_frobit. However, let's not touch mod_cache > and friends for now. > > We can rearrange things later if this new "architecture" actually has > any benefits. I am concerned that overgeneralization is going to make > things slower. So, I'd prefer to see us remove code rather than add; > but to also do it in parallel. So, I'd like to defer touching > mod_cache until we know we have something that is concretely better. -- +1. First have a working alternative to the current code and toss the current code if the new one is better (whatever better means by then exactly). Regards Rüdiger
Re: Generic cache architecture
On 05/03/2006 11:27 PM, William A. Rowe, Jr. wrote: > > Moreso, we need more third party authors to -participate- in telling us > what > in HTTPD-2.4 will make their module better. And a faster cycle of 6mos-1yr > gives them a chance to do this and realize the benefits in the official > release more quickly. But some of them will fall off the shelf (I guess especially some commercial ones), because they do not want to invest time again into an changing API. This means that they stick with an older version and do not do releases for each major version, but lets say for every second. Furthermore having frequent major releases increases the backport requests from user side and thus likely the backport efforts, as some users stick with older versions for whatever reasons (e.g. third party modules). I know we had this discussion about major release cycles several times in the past. Both approaches have pros and cons. So each side can dig out its old arguments / mails as I do not think that something essential new has been added to the pros and cons since last time :-). Maybe we should have a FDT (Frequently Discussed Topics) to collect the arguments ;-). > > Note that 2.2 will be a year old by December, so even if this concerns you, > we are already half way there. One for you. In 5 month httpd 2.2 is not far away from its first birthday. I hate infant mortality :-). Regards Rüdiger
Re: Generic cache architecture
On 5/3/06, Roy T. Fielding <[EMAIL PROTECTED]> wrote: However, I see no reason to start by changing the existing module names and assuming that one cache fits all. For simplicity sake, I agree. Let's call this new thing mod_cache_generic or mod_frobit. However, let's not touch mod_cache and friends for now. We can rearrange things later if this new "architecture" actually has any benefits. I am concerned that overgeneralization is going to make things slower. So, I'd prefer to see us remove code rather than add; but to also do it in parallel. So, I'd like to defer touching mod_cache until we know we have something that is concretely better. -- justin
Re: Third party modules for Apache2.x
At 00:20 04.05.2006, you wrote: Are you asking an httpd project related question? Not directly. If the httpd source code isn't your question, or the features it provides, then perhaps [EMAIL PROTECTED] (email apache-modules-subscribe to hop on) is a better forum as an "httpd module developers/user list". OK. I give [EMAIL PROTECTED] a try. Maybe it is better to ask there. Be happy to help you here or at apache-modules. OK, thanks. Sierk Sierk Bornemann | Hannover | Germany ICQ:221105136 e-mail: [EMAIL PROTECTED] URL: http://sierkbornemann.de/
Re: apr_brigade_insert_file() LFS/Linux issues
On 05/03/2006 11:58 PM, Joshua Slive wrote: >> >> No, works fine here. Can you get strace output for the failing call? > > > Just a random note: There was a report of a problem with 2.2.2 and > mod_include on the users list that went away when sendfile was turned > off. Sounds similar. Is this the same as PR 39463 (http://issues.apache.org/bugzilla/show_bug.cgi?id=39463)? I am not quite sure if these are similar to each other, because in 39463 (on Solaris 10) there seems to be an issue with sendfilev64, whereas in Niklas case I get the feeling that sendfile is used instead of its 64 bit version. Regards Rüdiger
Re: RFC: rename mod_cache to mod_http_cache
On 5/3/06, Paul Querna <[EMAIL PROTECTED]> wrote: I am okay with forcing people to wait for 2.4. Develop in trunk and/or devel-branches freely. Don't worry about back porting it to the stable branch, IMO. +1. -- justin
Re: Third party modules for Apache2.x
Sierk Bornemann wrote: Hi! Is this the right place to ask and discuss third party modules for Apache2.x? I have some questions concerning mod_tidy (http://mod-tidy.sourceforge.net/), especially running under Apache2.2.x/Win32. Well that depends. Are you asking an httpd project related question? if so by all means, ask here! If the httpd source code isn't your question, or the features it provides, then perhaps [EMAIL PROTECTED] (email apache-modules-subscribe to hop on) is a better forum as an "httpd module developers/user list". Is there maybe sombody out there, who eventually is able to give feedback about running the module, especially running it under Win32? Be happy to help you here or at apache-modules.
Third party modules for Apache2.x
Hi! Is this the right place to ask and discuss third party modules for Apache2.x? I have some questions concerning mod_tidy (http://mod-tidy.sourceforge.net/), especially running under Apache2.2.x/Win32. Is there maybe sombody out there, who eventually is able to give feedback about running the module, especially running it under Win32? Sierk Bornemann Core developer and project admin mod_tidy project Sierk Bornemann | Hannover | Germany ICQ:221105136 e-mail: [EMAIL PROTECTED] URL: http://sierkbornemann.de/
Re: apr_brigade_insert_file() LFS/Linux issues
On 5/3/06, Joe Orton <[EMAIL PROTECTED]> wrote: On Wed, May 03, 2006 at 02:39:33PM +0200, Niklas Edmundsson wrote: > I've run into apr_brigade_insert_file() creating brigades that's not > possible to sendfile() (EINVAL), this is with httpd-2.2.2 on Ubuntu > Breezy Linux amd64 (64bit). The file in question is 4.3GB, and it > seems that sendfile() doesn't cope with that. > > Has anyone else seen this? No, works fine here. Can you get strace output for the failing call? Just a random note: There was a report of a problem with 2.2.2 and mod_include on the users list that went away when sendfile was turned off. Sounds similar. Joshua.
Re: Possible new cache architecture
On 5/3/06, Graham Leggett <[EMAIL PROTECTED]> wrote: Gonzalo Arana wrote: > again, I am in the dark: why do cache request headers may need to be > replaced or edited in the same entity? It's a requirement of the HTTP/1.1 spec. non-modified response headers to conditional requests need to update cached response headers. we should try to avoid 'dialog' with cache backend. The catch is when the server sent "304 Not Modified" - you need to update your cache to say "yep, my cached entry is still fresh", ie update the headers, without touching the body, which hasn't changed. I see the light now :). Having a single cache_admin proc/thread would make this easier, since any operation can be presented as atomic, while it may require more than a single syscall (I know, the goal is avoid full entity duplication). Anyway, I guess a good policy is to have 'editable' content as binary data (i.e., no variable length). Perhaps this is not possible anyway :(. Of course, to avoid a 'dialog' between httpd process and cache_admin, both cache_admin and httpd must be smart enough. > That's why I suggested a dedicated process/thread for cache > administration, which is not a good idea if too many lookups are > issued to this process on each request received. I think in the long run, a dedicated process is the way to go. +1 :). Regards, -- Gonzalo A. Arana
Re: Generic cache architecture
Graham Leggett wrote: Ruediger Pluem wrote: Please keep in mind that some of us are dependent on commercial httpd modules, whether we like it or not. If the major upgrades happen in cyles shorter than a year I guess it is hard to get the commercial vendors to provide them. Not everybody is that innovative and fast as the ASF :-). +1. -0. You forget that we were frequently breaking the API way-way-back-when, and the good vendors kept up, and the lousy ones didn't. Moreso, we need more third party authors to -participate- in telling us what in HTTPD-2.4 will make their module better. And a faster cycle of 6mos-1yr gives them a chance to do this and realize the benefits in the official release more quickly. Note that 2.2 will be a year old by December, so even if this concerns you, we are already half way there. Bill
Re: Generic cache architecture
Roy T. Fielding wrote: A front-end cache is a completely different beast from a back-end cache. It doesn't make any sense to me to try to make them the same, and it certainly isn't elegant. SSL session, ldap credentials, sessions, and all those related things are trivial memory blocks that *are* suitable for back-end caching. /nod - I can appreciate the distinction. That said, the we will be better for grouping front end cache providers together, with one solid reference implementation (play your optimization games in experimental providers), and likewise a framework for the back end cache providers with one solid reference implementation, and I suspect httpd gets a whole lot more simple and stable with these available to third party authors. Bill
Re: Generic cache architecture
Gonzalo Arana wrote: On 5/3/06, Brian Akins <[EMAIL PROTECTED]> wrote: Is anyone else interested in having a generic cache architecture? (not http). I have plenty of cases were I re-invent the wheel for caching various things (IP's, sessions, whatever, etc.). It would be nice to have a provider based architecture for such things. I am. How about adding it to apr? 1. this isn't the [EMAIL PROTECTED] list, so your inquiry is 1/2 off topic 2. apr isn't a dumping ground 3. however... to the extent that this really portably solves the backend storage problem through an array of different providers, that well fits into apr-util's mission. A *vanilla* data store. The caching mechanics of request bodies will always belong in httpd.
Re: Generic cache architecture
> > Let's talk about httpd. We have a cache of ssl sessions. We have > a cache > of httpd response bodies. We have a cache of ldap credentials. A > really > thorough mod_usertrack would have a cache of user sessions. > > So really, it doesn't make sense to have these four wheels spinning > out of > sync at different stages of stability and performance. I'm > strongly +1 to > provide this functionality once, and reuse. On the contrary, it makes no sense whatsoever to use a generic storage facility for cached HTTP responses in a front-end cache because those responses can only be delivered at maximum speed through a single system call IFF they are not generic. That is why our front-end cache is not, and has never needed to be, a generic cache. I have to disagree: indeed a single syscall implies maximum speed & minimum memcpy (kernel to user, user to kernel), but consider that a cached response perhaps needs to get compressed (Transfer-Encoding: gzip, for instance). So, there is no way to assure that a single syscall will work. A generic cache, if designed with propper care, could provide a filedescriptor, which can be used with sendfile(2) or mmap(2)/write(2)/munmap(2) or any other combination. A front-end cache is a completely different beast from a back-end cache. It doesn't make any sense to me to try to what do you mean by 'front end cache' and 'backend cache'? make them the same, and it certainly isn't elegant. SSL session, ldap credentials, sessions, and all those related things are trivial memory blocks that *are* suitable for back-end caching. I have no objection to creating a module for back-end caching. I have no objection to creating five different forms of caching modules, each with its own qualities, that can be selected by configuration (preferably based on some suggested site profile). perhaps each kind of cache could be used by different parts (SSL session, ldap credentials, session, would use the 'backend cache'), and HTTP would use 'front-end cache'. However, I see no reason to start by changing the existing module names and assuming that one cache fits all. Regards, -- Gonzalo A. Arana
Re: Possible new cache architecture
On 05/03/2006 10:46 PM, Graham Leggett wrote: > > mod_cache definitely needs cache admin, currently it's implemented as an > external program that is called via cron, which doesn't help if you're > on a box without cron. Cache cleaning can be done either when a Not completely true. According to the documentation you can start it as a daemon (-d ,http://httpd.apache.org/docs/2.2/programs/htcacheclean.html#options) that runs periodically. Of course this daemon has to be started and configured separately from httpd, so it may not be the final solution. Regards Rüdiger
Re: Generic cache architecture
Ruediger Pluem wrote: Please keep in mind that some of us are dependent on commercial httpd modules, whether we like it or not. If the major upgrades happen in cyles shorter than a year I guess it is hard to get the commercial vendors to provide them. Not everybody is that innovative and fast as the ASF :-). +1. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Generic cache architecture
Roy T. Fielding wrote: > provide this functionality once, and reuse On the contrary, it makes no sense whatsoever to use a generic storage facility for cached HTTP responses in a front-end cache because those responses can only be delivered at maximum speed through a single system call IFF they are not generic. That is why our front-end cache is not, and has never needed to be, a generic cache. a generic cache can deliver objects in a single system call. Thinks VFS. the "generic storage facility" may be only a thin wrapper around something like current mod_disk_cache or it may be a memcache frontend, or something completely different. Trust me, I am extremely concerned about performance. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Possible new cache architecture
Gonzalo Arana wrote: again, I am in the dark: why do cache request headers may need to be replaced or edited in the same entity? It's a requirement of the HTTP/1.1 spec. HTTP requests can be conditional, in other words a browser (or a proxy) can ask a server "give me this URL, but only if it has changed from my cached copy". If the server thinks that the file has changed (or Cache-Control: no-cache was specified), then the server will send a full response back headers + body, and the browser/proxy replaces it's cached copy with the new headers+body. If the server thinks that the file is the same, ie it didn't change, the server sends back the magic code "304 Not Modified", and just the headers - without any body. These new headers must replace the existing headers in the browser/proxy's cached entry, making the cached entry "fresh" again. And here lies the problem. Doing the request this way means you don't have to ask the backend "is my cached copy still fresh?", get an answer back "No", and then send a second request saying "ok then, give me the new data" - you can implement caching in one request. The catch is when the server sent "304 Not Modified" - you need to update your cache to say "yep, my cached entry is still fresh", ie update the headers, without touching the body, which hasn't changed. That's why I suggested a dedicated process/thread for cache administration, which is not a good idea if too many lookups are issued to this process on each request received. mod_cache definitely needs cache admin, currently it's implemented as an external program that is called via cron, which doesn't help if you're on a box without cron. Cache cleaning can be done either when a connection is complete in the existing process (which may be simpler to implement, but it runs after every connection), or it can be done as you suggest, where a dedicated thread/process handles this independently. I think in the long run, a dedicated process is the way to go. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Adding my own Handler in Apache
Have a look at mod_example.c (bundled with httpd). It will give you a quick idea on how modules are structured. Also, read apxs instructions, it can generate a 'wireframe' for your apache module, compile it, link it and even install it. Regards, On 5/3/06, Tiago Semprebom <[EMAIL PROTECTED]> wrote: Hello, I created my own Handler, called mod_mytest.c (this file was alterated from the send-as-is), In the http.conf file I set the AddHandler and SetHandler : 1) How I compile this new handler (mod_mytest.c)? 2) I need to change more files to do it work? Thank's in advanced, Tiago Semprebom -- Gonzalo A. Arana
Re: Generic cache architecture
On May 3, 2006, at 12:53 PM, William A. Rowe, Jr. wrote: Brian Akins wrote: Is anyone else interested in having a generic cache architecture? (not http). I have plenty of cases were I re-invent the wheel for caching various things (IP's, sessions, whatever, etc.). It would be nice to have a provider based architecture for such things. Let's talk about httpd. We have a cache of ssl sessions. We have a cache of httpd response bodies. We have a cache of ldap credentials. A really thorough mod_usertrack would have a cache of user sessions. So really, it doesn't make sense to have these four wheels spinning out of sync at different stages of stability and performance. I'm strongly +1 to provide this functionality once, and reuse. On the contrary, it makes no sense whatsoever to use a generic storage facility for cached HTTP responses in a front-end cache because those responses can only be delivered at maximum speed through a single system call IFF they are not generic. That is why our front-end cache is not, and has never needed to be, a generic cache. A front-end cache is a completely different beast from a back-end cache. It doesn't make any sense to me to try to make them the same, and it certainly isn't elegant. SSL session, ldap credentials, sessions, and all those related things are trivial memory blocks that *are* suitable for back-end caching. I have no objection to creating a module for back-end caching. I have no objection to creating five different forms of caching modules, each with its own qualities, that can be selected by configuration (preferably based on some suggested site profile). However, I see no reason to start by changing the existing module names and assuming that one cache fits all. Roy
Adding my own Handler in Apache
Hello,I created my own Handler, called mod_mytest.c (this file was alterated from the send-as-is), In the http.conf file I set the AddHandler and SetHandler :1) How I compile this new handler (mod_mytest.c)?2) I need to change more files to do it work?Thank's in advanced,Tiago Semprebom Novidade no Yahoo! Mail: receba alertas de novas mensagens no seu celular. Registre seu aparelho agora!
Re: Generic cache architecture
On 05/03/2006 09:53 PM, William A. Rowe, Jr. wrote: > > And finally (most important) none of this needs to target 2.2. If 2.2 > lives > 5 months to be replaced by 2.4 - there is really no issue. 2.0 lived Please keep in mind that some of us are dependent on commercial httpd modules, whether we like it or not. If the major upgrades happen in cyles shorter than a year I guess it is hard to get the commercial vendors to provide them. Not everybody is that innovative and fast as the ASF :-). Regards Rüdiger
Re: Possible new cache architecture
Roy T. Fielding wrote: That is a heck of a lot easier than convincing everyone to dump the current code based on an untested theory. I think the idea may be a lot more tested than you think. Most things I "suggest" have had an incubation period somewhere... I'm fine with not screwing with current mod_cache. I just think it should be either: renamed or made generic. We may or may not need a generic mod_backend_cache. I have posted a "psuedo-implementation" that got lost in the latest thread bloat. I can repost if anyone is interested. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Generic cache architecture
Gonzalo Arana wrote: I am. How about adding it to apr? How about someone figuring out how to get providers into apr? Doesn't look horribly hard. Perhaps I should ask on apr-devel? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Generic cache architecture
Brian Akins wrote: Is anyone else interested in having a generic cache architecture? (not http). I have plenty of cases were I re-invent the wheel for caching various things (IP's, sessions, whatever, etc.). It would be nice to have a provider based architecture for such things. Let's talk about httpd. We have a cache of ssl sessions. We have a cache of httpd response bodies. We have a cache of ldap credentials. A really thorough mod_usertrack would have a cache of user sessions. So really, it doesn't make sense to have these four wheels spinning out of sync at different stages of stability and performance. I'm strongly +1 to provide this functionality once, and reuse. While we are at it, the proxy backend requester should be generic enough that if I need to fetch, say, a trusted CA reference from a backend (dmz) server, that code shouldn't be rewritten either. But it's not oriented to the request so it would be good to see some more modularity on the proxy backend while we improve the cache middle layer. And finally (most important) none of this needs to target 2.2. If 2.2 lives 5 months to be replaced by 2.4 - there is really no issue. 2.0 lived too long because 2.1.x stayed in flux to long. Bill
Re: Generic cache architecture
On 5/3/06, Brian Akins <[EMAIL PROTECTED]> wrote: Is anyone else interested in having a generic cache architecture? (not http). I have plenty of cases were I re-invent the wheel for caching various things (IP's, sessions, whatever, etc.). It would be nice to have a provider based architecture for such things. I am. How about adding it to apr? Regards, -- Gonzalo A. Arana
Re: Possible new cache architecture
Thanks for bringing me to the light. On 5/3/06, Graham Leggett <[EMAIL PROTECTED]> wrote: Gonzalo Arana wrote: > Excuse my ignorance in this matter, but about the 'cache sub-key' > issue, why not just use a generic cache (with some expiration model > -LRU, perhaps-) with a 'smart' comparison function? So far one of the best suggestions was from the patch posted recently, where the headers and body were in the same file, but where the headers were given "breathing room" before the cache body, so that the headers can be replaced (within reasonable limits). What this means is that each key/data entry is now a single file again (like in 1.3), which is much easier to clean up atomically. The problem still remains that an existing cache file's headers must be editable, without doing expensive operations like copying, and this again, I am in the dark: why do cache request headers may need to be replaced or edited in the same entity? editing must be atomic (no use one thread/process trying to serve content from the cache and halfway through, another thread tries to update the headers). This will require some form of locking, which may be too much of a performance drag, thus blowing the back-to-one-file idea out the water. this makes sense, but I still do not understand the origin of the problem (in-place header replacement). Problems with cache expiry though are a real problem that mod_cache suffers from now, and need to be fixed. That's why I suggested a dedicated process/thread for cache administration, which is not a good idea if too many lookups are issued to this process on each request received. Regards, -- Gonzalo A. Arana
Generic cache architecture
Is anyone else interested in having a generic cache architecture? (not http). I have plenty of cases were I re-invent the wheel for caching various things (IP's, sessions, whatever, etc.). It would be nice to have a provider based architecture for such things. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: RFC: rename mod_cache to mod_http_cache
Graham Leggett wrote: > Brian Akins wrote: > >> Not wanting to stir the huge pot o' stuff that is going on here, but >> what are the thoughts of renaming mod_cache to mod_http_cache? >> mod_cache is http specific. This would follow the general ide that >> mod_proxy uses. > > This is a good idea, but thinking about this for a bit, doing so would > be impossible to backport to v2.2 (it would break existing configs). > This in turn would make it more difficult for fixes that would be useful > in v2.2 to be backported, forcing people to wait until v2.4 before > seeing the advantages. I am okay with forcing people to wait for 2.4. Develop in trunk and/or devel-branches freely. Don't worry about back porting it to the stable branch, IMO. -Paul
Re: RFC: rename mod_cache to mod_http_cache
William A. Rowe, Jr. wrote: Not in 2.2 branch, but in trunk? The issue is that it's half httpd, and half generic. Let me mull this over. can we separate out the http specific parts without violating Graham's concerns? My whole original idea was to just do that... I was not fully aware of the issues in the current mod_cache. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: RFC: rename mod_cache to mod_http_cache
Brian Akins wrote: Not wanting to stir the huge pot o' stuff that is going on here, but what are the thoughts of renaming mod_cache to mod_http_cache? mod_cache is http specific. This would follow the general ide that mod_proxy uses. This is a good idea, but thinking about this for a bit, doing so would be impossible to backport to v2.2 (it would break existing configs). This in turn would make it more difficult for fixes that would be useful in v2.2 to be backported, forcing people to wait until v2.4 before seeing the advantages. I agree that a rename should definitely happen though, but I'd say not yet. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: RFC: rename mod_cache to mod_http_cache
Brian Akins wrote: Not wanting to stir the huge pot o' stuff that is going on here, but what are the thoughts of renaming mod_cache to mod_http_cache? mod_cache is http specific. This would follow the general ide that mod_proxy uses. I am not suggesting changing any functionality at this time, simply renaming it to a more suitable name. Not in 2.2 branch, but in trunk? The issue is that it's half httpd, and half generic. Let me mull this over. If we layer it, I can entirely agree that there should be a mod_http_cache that's entirely concerned with the content negotation handshake of http. But in all other respects, mod_cache is equally useful for other protocols such as mod_ftp (it takes advantage of it now, only with one possible variant because ftp doesn't speak in variants.) Bill
RFC: rename mod_cache to mod_http_cache
Not wanting to stir the huge pot o' stuff that is going on here, but what are the thoughts of renaming mod_cache to mod_http_cache? mod_cache is http specific. This would follow the general ide that mod_proxy uses. I am not suggesting changing any functionality at this time, simply renaming it to a more suitable name. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Possible new cache architecture
Roy T. Fielding wrote: For the record, Graham's statements were entirely correct, Brian's suggested architecture would slow the HTTP cache, No. It would simplify the existing implementation. The existing implementation, as Graham has noted, is not "fully functional." Graham argues - and I'm still mulling it over - that a generic cache architecture would get in the way of making a fully functional http cache. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Possible new cache architecture
Gonzalo Arana wrote: Excuse my ignorance in this matter, but about the 'cache sub-key' issue, why not just use a generic cache (with some expiration model -LRU, perhaps-) with a 'smart' comparison function? So far one of the best suggestions was from the patch posted recently, where the headers and body were in the same file, but where the headers were given "breathing room" before the cache body, so that the headers can be replaced (within reasonable limits). What this means is that each key/data entry is now a single file again (like in 1.3), which is much easier to clean up atomically. The problem still remains that an existing cache file's headers must be editable, without doing expensive operations like copying, and this editing must be atomic (no use one thread/process trying to serve content from the cache and halfway through, another thread tries to update the headers). This will require some form of locking, which may be too much of a performance drag, thus blowing the back-to-one-file idea out the water. Problems with cache expiry though are a real problem that mod_cache suffers from now, and need to be fixed. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Possible new cache architecture
On Wed, 3 May 2006 11:39:02 -0700 "Roy T. Fielding" <[EMAIL PROTECTED]> wrote: > On May 3, 2006, at 5:56 AM, Davi Arnaut wrote: > > > On Wed, 3 May 2006 14:31:06 +0200 (SAST) > > "Graham Leggett" <[EMAIL PROTECTED]> wrote: > > > >> On Wed, May 3, 2006 1:26 am, Davi Arnaut said: > >> > Then you will end up with code that does not meet the > requirements of > HTTP, and you will have wasted your time. > >>> > >>> Yeah, right! How ? Hey, you are using the Monty Python argument > >>> style. > >>> Can you point to even one requirement of HTTP that my_cache_provider > >>> wont meet ? > >> > >> Yes. Atomic insertions and deletions, the ability to update headers > >> independantly of body, etc etc, just go back and read the thread. > > > > I can't argue with a zombie, you keep repeating the same > > misunderstands. > > > >> Seriously, please move this off list to keep the noise out of > >> people's > >> inboxes. > > > > Fine, I give up. > > For the record, Graham's statements were entirely correct, > Brian's suggested architecture would slow the HTTP cache, > and your responses have been amazingly childish for someone > who has earned zero credibility on this list. Fine, I do have zero credibility. > I suggest you stop defending a half-baked design theory and > just go ahead and implement something as a patch. If it works, > that's great. If it slows the HTTP cache, I will veto it myself. I'm already doing this. > There is, of course, no reason why the HTTP cache has to use > some new middle-layer back-end cache, so maybe you could just > stop arguing about vaporware and simply implement a single > mod_backend_cache that doesn't try to be all things to all people. > > Implement it and then convince people on the basis of measurements. > That is a heck of a lot easier than convincing everyone to dump > the current code based on an untested theory. > I just wanted to get comments (the original idea wasn't mine). It wasn't my intention to flame anyone, I'm not mad or anything. I was just stating my opinion. I maybe wrong, but I don't give up easy. :) -- Davi Arnaut
Re: Possible new cache architecture
On May 3, 2006, at 5:56 AM, Davi Arnaut wrote: On Wed, 3 May 2006 14:31:06 +0200 (SAST) "Graham Leggett" <[EMAIL PROTECTED]> wrote: On Wed, May 3, 2006 1:26 am, Davi Arnaut said: Then you will end up with code that does not meet the requirements of HTTP, and you will have wasted your time. Yeah, right! How ? Hey, you are using the Monty Python argument style. Can you point to even one requirement of HTTP that my_cache_provider wont meet ? Yes. Atomic insertions and deletions, the ability to update headers independantly of body, etc etc, just go back and read the thread. I can't argue with a zombie, you keep repeating the same misunderstands. Seriously, please move this off list to keep the noise out of people's inboxes. Fine, I give up. For the record, Graham's statements were entirely correct, Brian's suggested architecture would slow the HTTP cache, and your responses have been amazingly childish for someone who has earned zero credibility on this list. I suggest you stop defending a half-baked design theory and just go ahead and implement something as a patch. If it works, that's great. If it slows the HTTP cache, I will veto it myself. There is, of course, no reason why the HTTP cache has to use some new middle-layer back-end cache, so maybe you could just stop arguing about vaporware and simply implement a single mod_backend_cache that doesn't try to be all things to all people. Implement it and then convince people on the basis of measurements. That is a heck of a lot easier than convincing everyone to dump the current code based on an untested theory. Roy
Re: Possible new cache architecture
Excuse my ignorance in this matter, but about the 'cache sub-key' issue, why not just use a generic cache (with some expiration model -LRU, perhaps-) with a 'smart' comparison function? We could use as key full request headers (perhaps somewhat parsed), and as a comparison function a clever enough code to handle Vary, entity aging and so on. Best regards, -- Gonzalo A. Arana
Re: Possible new cache architecture
Brian Akins wrote: Does this discussion belong off-list? I would think this is the type of thing we need to discuss on this list. The technical discussion belongs on the list, flames not. Is there any consensus as to how to move forward? Do we just leave it as it is currently? There is a patch on the table, let's review it. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Possible new cache architecture
William A. Rowe, Jr. wrote: --1. This is a development list. If you don't want development discussions, don't subscribe. I was referring to the flamebait, development discussions would obviously remain on the list. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Possible new cache architecture
Brian Akins wrote: Moving towards and keeping with the above goals is a far higher priority than simplifying the generic backend cache interface. This response was a perfect summation of why we do *not* run the stock mod_cache here... Having the source means you can customise and improve the code to better meet your needs, and in your case your modifications work for you, and your organisation has the resources to commission and maintain those modifications. The trouble is, in order to be accepted into httpd, your modifications have to work for everyone else as well. Apparently for example the problem of trying to handle subkeys under a main key "is mod_http_cache's problem". Ok, so mod_httpd_cache now has to implement locking mechanisms to try and somehow turn the elegant (but overly simplistic) mod_cache into a cache that is practically useful. In the process we slow the cache down. The whole point of the cache is to speed things up. Suddenly, we lose the whole point of the exercise. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Possible new cache architecture
Graham Leggett wrote: Seriously, please move this off list to keep the noise out of people's inboxes. --1. This is a development list. If you don't want development discussions, don't subscribe. Bill
Re: Possible new cache architecture
Graham Leggett wrote: Seriously, please move this off list to keep the noise out of people's inboxes. Does this discussion belong off-list? I would think this is the type of thing we need to discuss on this list. Is there any consensus as to how to move forward? Do we just leave it as it is currently? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Possible new cache architecture
Graham Leggett wrote: Moving towards and keeping with the above goals is a far higher priority than simplifying the generic backend cache interface. This response was a perfect summation of why we do *not* run the stock mod_cache here... -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: apr_brigade_insert_file() LFS/Linux issues
On Wed, 3 May 2006, Joe Orton wrote:
I've run into apr_brigade_insert_file() creating brigades that's not
possible to sendfile() (EINVAL), this is with httpd-2.2.2 on Ubuntu
Breezy Linux amd64 (64bit). The file in question is 4.3GB, and it
seems that sendfile() doesn't cope with that.
Has anyone else seen this?
No, works fine here. Can you get strace output for the failing call?
[pid 931] open("/httpcache/HU/zG/8j/_HJBmSBIt0F2CnvQ", O_RDONLY) = 11
...
[pid 931] sendfile(10, 11, [4096], 4571090944) = -1 EINVAL (Invalid argument)
And with the file being 4571095040 bytes, that offset and len seems
correct to me...
And just to be sure it isn't my patch, the same without mod_disk_cache
loaded:
[pid 5746]
open("/export/ftp/mirror/media/StarWars-Revelations/revelations-2.iso",
O_RDONLY) = 11
...
[pid 5746] sendfile(10, 11, [0], 4571090944) = -1 EINVAL (Invalid argument)
It should be noted that in the latter case apr_brigade_insert_file()
isn't even used since server/core.c hasn't been converted to use it.
This seems to be some sendfile/sendfile64 mess, any ideas on how to
approach it?
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
"Hell has no fury as a woman scorned" - Deanna Troi
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: apr_brigade_insert_file() LFS/Linux issues
On Wed, May 03, 2006 at 02:39:33PM +0200, Niklas Edmundsson wrote: > I've run into apr_brigade_insert_file() creating brigades that's not > possible to sendfile() (EINVAL), this is with httpd-2.2.2 on Ubuntu > Breezy Linux amd64 (64bit). The file in question is 4.3GB, and it > seems that sendfile() doesn't cope with that. > > Has anyone else seen this? No, works fine here. Can you get strace output for the failing call? joe
Re: r382799 - /httpd/httpd/trunk/modules/ssl/ssl_scache_shmcb.c
On Sun, Mar 05, 2006 at 01:21:08AM -0500, Geoff Thorpe wrote: > On March 3, 2006 08:11 am, Joe Orton wrote: > [snip] > > Log: > > * modules/ssl/ssl_scache_shmcb.c (shmcb_safe_clear): Mark with > > "noinline" attribute for GCC > 3. > [snip] > > This reminded me that I had a rewrite of shmcb lurking around that needed > dusting off and updating due to changes in CVS. I've attached a new > version that is a drop-in replacement for 2.2.0 - I think it addresses > all the "safe" memcpy issues and would *finally* allow Ken Coar to remove > that shmcb item from STATUS. It also obviates the need for the gcc-fear > your commit aptly illustrated :-) (The new version is no longer > vulnerable to agressive memset optimisation - the only memsets should be > for binary data). Geoff - sorry for the ultra-slow response. This looks great, much simpler code to follow. I've committed this to the trunk with a few minor tweaks: http://svn.apache.org/viewcvs.cgi?rev=399291&view=rev Thanks a lot! joe
Re: Possible new cache architecture
On Wed, 3 May 2006 14:31:06 +0200 (SAST) "Graham Leggett" <[EMAIL PROTECTED]> wrote: > On Wed, May 3, 2006 1:26 am, Davi Arnaut said: > > >> Then you will end up with code that does not meet the requirements of > >> HTTP, and you will have wasted your time. > > > > Yeah, right! How ? Hey, you are using the Monty Python argument style. > > Can you point to even one requirement of HTTP that my_cache_provider > > wont meet ? > > Yes. Atomic insertions and deletions, the ability to update headers > independantly of body, etc etc, just go back and read the thread. I can't argue with a zombie, you keep repeating the same misunderstands. > Seriously, please move this off list to keep the noise out of people's > inboxes. Fine, I give up. -- Davi Arnaut
apr_brigade_insert_file() LFS/Linux issues
Hi all!
I've run into apr_brigade_insert_file() creating brigades that's not
possible to sendfile() (EINVAL), this is with httpd-2.2.2 on Ubuntu
Breezy Linux amd64 (64bit). The file in question is 4.3GB, and it
seems that sendfile() doesn't cope with that.
Has anyone else seen this?
Quick testing without apr involved shows that by just including
sys/sendfile.h without any defines I get the old sendfile, if I define
_FILE_OFFSET_BITS=64 sendfile gets redefined as sendfile64...
I somehow assumed that you would get LFS-stuff by default on a 64bit
platform, but it doesn't seem to be that simple ;)
/Nikke - slightly confused
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
"This station is now the ultimate power in the universe." - Admiral Motti
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: Possible new cache architecture
On Wed, May 3, 2006 1:26 am, Davi Arnaut said: >> Then you will end up with code that does not meet the requirements of >> HTTP, and you will have wasted your time. > > Yeah, right! How ? Hey, you are using the Monty Python argument style. > Can you point to even one requirement of HTTP that my_cache_provider > wont meet ? Yes. Atomic insertions and deletions, the ability to update headers independantly of body, etc etc, just go back and read the thread. Seriously, please move this off list to keep the noise out of people's inboxes. Regards, Graham --
Re: Possible new cache architecture
> -Ursprüngliche Nachricht- > Von: Joe Orton > > The way I would expect it to work would be by passing f->next > in to the > store_body callback, it looks doomed to eat RAM as currently > designed. > mod_disk_cache's store_body implementation can then do: > > 1. read bucket(s) from brigade, appending to some temp brigade > 2. write bucket(s) in temp brigade to cache file > 3. pass temp brigade on to f->next > 4. clear temp brigade to ensure memory is released > 5. goto 1 Yes, this was also my idea, but I would like to avoid this, because: 1. This is an API change which might be hard to backport. 2. I do not really like the close tie between the storage provider and the filter chain. It forces the provider to do things it should not care about from my point of view. Furthermore: What about mod_cache in this case? Do you want to skip ap_pass_brigade there or do you want to cleanup the original brigade inside store_body of mod_disk_cache and let mod_cache pass an empty brigade up the chain? If we decide to skip ap_pass_brigade inside mod_cache all storage providers need to ensure that they pass the data up the chain which seems duplicated code to me and does not seem to belong to their core tasks. OTH doing this in mod_cache and only pass the small brigade to store_body of the provider has the drawback that mod_mem_cache wants to see the original file buckets in order to save the file descriptors of the files. To be honest, currently I have no solution at hand that I really like, but I agree that this really needs to be changed. Regards Rüdiger
Add new Handler/Filter in Apache 2.0
I need to insert a new Handler/Filter for manipulate requests in Apache 2.0, How I can do It ?Thank's in advanced Tiago Semprebom Abra sua conta no Yahoo! Mail - 1GB de espaço, alertas de e-mail no celular e anti-spam realmente eficaz.
Re: Possible new cache architecture
On Tue, May 02, 2006 at 02:21:27PM +0200, Plüm, Rüdiger, VF EITO wrote: > Another thing: I guess on systems with no mmap support the current > implementation > of mod_disk_cache will eat up a lot of memory if you cache a large local file, > because it transforms the file bucket(s) into heap buckets in this case. > Even if mmap is present I think that mod_disk_cache causes the file buckets > to be transformed into many mmap buckets if the file is large. Thus we do not > use sendfile in the case we cache the file. > I the case that a brigade only contains file_buckets it might be possible to > "copy" this brigade, sent it up the chain and process the copy of the brigade > for disk storage afterwards. Of course this opens a race if the file gets > changed in between these operations. > This approach does not work with socket or pipe buckets for obvious reasons. > Even heap buckets seem to be a somewhat critical idea because of the added > memory usage. The way I would expect it to work would be by passing f->next in to the store_body callback, it looks doomed to eat RAM as currently designed. mod_disk_cache's store_body implementation can then do: 1. read bucket(s) from brigade, appending to some temp brigade 2. write bucket(s) in temp brigade to cache file 3. pass temp brigade on to f->next 4. clear temp brigade to ensure memory is released 5. goto 1 joe
Re: [Fwd: 2.2+ security page empty?]
>There is nothing on the security page any more for 2.2, is there a bug > with the report you use to populate it? Fixed Cheers, Mark
