On Tue, 27 Nov 2001, William A. Rowe, Jr. wrote: > Bzzt... wrong. There are plenty of non-zero copy operations in the > world... this isn't the 'unique exception' ... it will be frequent > enough to warrent redesign. > > > Perhaps a better solution would be to just create a read function > > that accepts a pre-allocated buffer that we must copy into. Then, > > filters that know they are going to throw the data away would incur > > the penalty of the copy in lieu of the penalty of the malloc(). > > With the file bucket, this has the advantage that the read from the > > disk can be put directly into the pre-allocated buffer, and the copy > > doesn't actually exist. > > Bingo. That way you haven't tied it up with specific bucket types - > but extended the api for all buckets.
Works for me. > This is where I keep saying that the consumer often knows more about > the mechanics of transfering or processing the data than the > producer... until we add some graceful methods for handling those > cases, kludges will abound. Well, here's another kludge for you: the problem is that apr_bucket_read() for file buckets falls into an abyss of performance as soon as the magic APR_MMAP_LIMIT is exceeded. That's because we give up on mmap entirely and read 8kb at a time frome the whole bigass file, even if all we wanted in the first place was the first 32 bytes or so (say for something like mod_mime_magic) and we're never going to touch the other 12MB. We still have to pay the penalty anyway. What if apr_bucket_read() on a file bucket bigger than APR_MMAP_LIMIT would turn the file bucket into an mmap bucket+file bucket pair, with the mmap bucket APR_MMAP_LIMIT in size, and the file bucket gets the rest? I think that would tackle this problem nicely. In a case like mod_mime_magic where we're only reading the beginning of the file, we'd actually still be able to use sendfile() with the first APR_MMAP_LIMIT's worth as a header. :) Of course, you might run into a case where you're os-limited as to how much you can have mmaped at one time per process if we buffer a large number of buckets for some reason, but then the mmap will fail and we'll fall back on read() [the code already does that]. It actually never occurred to me before that what happens with the code now on a file bigger than APR_MMAP_LIMIT is that we get a bunch of 8KB heap buckets until there is <= APR_MMAP_LIMIT _left_ in the file, and that last bit of the file is mmaped. So we have a bunch of 8KB heap buckets followed by a 4MB-ish mmap. Man, that's weird. :) --Cliff -------------------------------------------------------------- Cliff Woolley [EMAIL PROTECTED] Charlottesville, VA
