I understand now - one has to bear in mind the case where the amount of requested data of a similar size to the cache size. So as you said, the better solution is to use a multi-streamed client and have squid deal with the partial file objects. It would also require one of these clients to support http proxy use.
This seems like a widely useful feature - what would it take to get this on the squid devel wish list? Meanwhile I`m going to bother the aget author.
Cheers, Rod.
Henrik Nordstrom wrote:
On Fri, 17 Dec 2004, Rod Walker wrote:
Hi,
Of course I don`t know the inner workings of Squid but I thought the
problem would be that the file chunks would always be different depending
on how many streams the user chose. So even if Squid cached partial file
objects, it's unlikely the client would ask for exactly those blocks. For
example, a client downloads a file with 2 streams and then another client
with 3
2 streams: 1-500 & 501-1000 would result in 2 objects cached.
3 streams: 1-333,334-666,667-1000 - would these requests give cache hits?
Yes, if the cache is implemented correctly.
As for the squid-squid, squid-server multi-stream ... If the requested object will not be cached because it`s larger than the max cachable size, then I can see the problems you mention. But if it is to be cached then it could be written straight to disk, and read back in a single stream to the client.
True, but how fast should this be done? the fetching of the parts which we can not yet send to the client. And how do we guarantee we do not run out of disk space? Concider the case there may be several of these requests in parallell for different resources.
I guess you normally keep the objects in memory though. I guess my conceptual problem is that I`m talking about objects around 1GB, and you objects more like 1MB.
No, I am talking about large objects. For small objects there is no noticeable benefit for multi-streamed transfers as the majority of the overhead is the session setup.
I think I can protoype the system with single streams, in the knowledge that multi-stream is at least possible. Thanks a lot for your help.
Regards Henrik
-- Rod Walker +1 6042913051
