On Wed, 2007-01-31 at 21:57 +0800, Adrian Chadd wrote: > I've done a quick brain-dump about the current storage manager state. > > http://wiki.squid-cache.org/DevelopmentIdeas/StorageStuff > > I'd appreciate some comments before I run off and start coding this stuff > up..
I agree that read-parse-dump-write-read-parse-dump-write sequence is inefficient for message headers and other metadata. IIRC, we talked about optimizing this at least since 1998. Stored binary metadata has its drawbacks, but overall it is probably a win for a performance-sensitive proxy. For many other problems and changes you are talking about, I would try reworking MemBuf or providing a similar object that would allow higher-level code to "copy" and "concatenate" chunks of memory without the actual copy taking place. Visualize a MemBuf with an offset and subsize fields. Now add support for a chain of such buffers that looks like a single buffer for higher-level code. With such low-level support and perhaps vector I/O, most performance problems with the high level code doing to much copying would either disappear or would be much easier to fix (by using appropriate buffers instead of char* manipulations). As an added bonus, it may be possible to avoid a lot of the copying when parsing headers because string-based headers would be able to refer to portions of the original I/O buffer. Proper reference counting of true/allocated buffers would be required to keep overall memory consumption comparable to current Squid, of course. Finally, I am not sure I agree regarding storage decision making time. An optimized storage system (the interesting case) would probably buffer/merge small chunks and would probably not store object chunks sequentially on disk, so the issue of the total object size becomes unimportant. Needless to say, I believe this work should be done in Squid3.1 code base or later :-/. HTH, Alex.
