Re: adding content to the cache
On 05/07/2012 11:11 PM, anita.sivaku...@wipro.com wrote: > I tried the following request but it returned saying it is an invalid request. > > In Foo: > PUT http://myserver/file1.html HTTP/1.1 > HOST:localhost > Accept:*/* > Content-Type:text/html > > Where file1.html is present in myserver (apache). > > I get the error as: > HTTP/1.0 400 Bad Request > Server: squid/3.1.16 > Mime-Version: 1.0 > Date: Tue, 08 May 2012 10:29:51 GMT > Content-Type: text/html > Content-Length: 3192 > X-Squid-Error: ERR_INVALID_URL 0 > Vary: Accept-Language > Content-Language: en > X-Cache: MISS from squid.packet-pushers.net > Via: 1.0 squid.packet-pushers.net (squid/3.1.16) > Connection: close > > Am I missing something in how to write a PUT request? I am not sure what you meant by "In Foo" but your PUT request quoted above is missing a body. See RFC 2616 or google for PUT request examples. If you are using squidclient, then according to documentation, the command to send a PUT request containing Foo.txt file to the origin server located at Bar URL is squidclient -P Foo.txt Bar However, it is not clear what a PUT request gives you since Squid cannot cache PUT request bodies. HTH, Alex. > -Original Message- > From: Alex Rousskov [mailto:rouss...@measurement-factory.com] > Sent: 07 May 2012 21:19 > To: Anita Sivakumar (WT01 - GMT-Telecom Equipment) > Cc: squid-dev@squid-cache.org > Subject: Re: adding content to the cache > > On 05/07/2012 05:35 AM, anita wrote: >>> It sends an HTTP PUT request to Squid and reads the response. The PUT >>> request body is taken from the named file. You most likely do not need >>> this because you want to GET content from the fake server, and not PUT >>> content to the fake server. Squid does not cache request bodies. > > >> Just came across this post. Wondering how to use this feature for >> squidclient. >> Can we simply do a squidclient -P http://server/filename.html ? >> Where the filename.html is the actual object that I would need to be pushed >> into the squid cache? > > > IIRC, "squidclient -P Foo" specifies that the PUT request body should > come from a file named Foo. Squid does not cache request bodies. > > Squid caches responses. If you want Squid to cache the contents of file > Foo, you have to make Foo available on some web server. It is relatively > easy to set up a "fake" server that responds with Foo to all HTTP > requests (and runs on the same host as Squid), but you can also use > Apache httpd or another full-blown origin server. > > With Rock Store (v3.2 and up), it is also possible to add something to > the Squid disk cache without using HTTP (because the cache index and > queues are stored in shared memory accessible to any process with enough > permissions). However, the above approach with a fake server is easier > if you just want to cache a single file. > > Please note that I do not really understand all the complexities of the > overall problem you are trying to solve (the description of the problem > that you posted earlier was too convoluted for me) so I cannot recommend > a good solution or evaluate whether the above is the right path towards > that solution. I am just trying to answer a specific question instead. > > > HTH, > > Alex. > > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any attachments to > this message are intended for the exclusive use of the addressee(s) and may > contain proprietary, confidential or privileged information. If you are not > the intended recipient, you should not disseminate, distribute or copy this > e-mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should > check this email and any attachments for the presence of viruses. The company > accepts no liability for any damage caused by any virus transmitted by this > email. > > www.wipro.com
Re: adding content to the cache
tis 2012-05-08 klockan 10:33 +1200 skrev Amos Jeffries: > Further to this. The HTTP specs do make PUT request bodies cacheable > under the URL the request specifies. Squid just does not support that > edge case of HTTP/1.1 yet. It would be nice to get it implemented for > the next Squid series (3.3+) but its not clear yet whether this is the > best solution for you. Where in the specs do it say that the request body may be cached? I am not aware of any mentioning of caching of PUT request bodies. In addition this body lacks any server side meta-information. Additionally the server MAY apply processing on the stored entity. the 201/200/204 response to PUT also lacks most or all of the meta data. > and why does that machine not simply pass squid a GET request for the > URL to pre-cache the request for later use (no need to know what the > content is in advance of the request, just GET with a max-age=0 control > header)? Further to this, you can use cache_peer + cache_peer_access etc to define from where Squid shoult GET the response. It does not need to be the actual requested server. Regards Henrik
RE: adding content to the cache
Hi Alex, I tried the following request but it returned saying it is an invalid request. In Foo: PUT http://myserver/file1.html HTTP/1.1 HOST:localhost Accept:*/* Content-Type:text/html Where file1.html is present in myserver (apache). I get the error as: HTTP/1.0 400 Bad Request Server: squid/3.1.16 Mime-Version: 1.0 Date: Tue, 08 May 2012 10:29:51 GMT Content-Type: text/html Content-Length: 3192 X-Squid-Error: ERR_INVALID_URL 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from squid.packet-pushers.net Via: 1.0 squid.packet-pushers.net (squid/3.1.16) Connection: close Am I missing something in how to write a PUT request? Thanks. Regards, Anita -Original Message- From: Alex Rousskov [mailto:rouss...@measurement-factory.com] Sent: 07 May 2012 21:19 To: Anita Sivakumar (WT01 - GMT-Telecom Equipment) Cc: squid-dev@squid-cache.org Subject: Re: adding content to the cache On 05/07/2012 05:35 AM, anita wrote: >> It sends an HTTP PUT request to Squid and reads the response. The PUT >> request body is taken from the named file. You most likely do not need >> this because you want to GET content from the fake server, and not PUT >> content to the fake server. Squid does not cache request bodies. > Just came across this post. Wondering how to use this feature for > squidclient. > Can we simply do a squidclient -P http://server/filename.html ? > Where the filename.html is the actual object that I would need to be pushed > into the squid cache? IIRC, "squidclient -P Foo" specifies that the PUT request body should come from a file named Foo. Squid does not cache request bodies. Squid caches responses. If you want Squid to cache the contents of file Foo, you have to make Foo available on some web server. It is relatively easy to set up a "fake" server that responds with Foo to all HTTP requests (and runs on the same host as Squid), but you can also use Apache httpd or another full-blown origin server. With Rock Store (v3.2 and up), it is also possible to add something to the Squid disk cache without using HTTP (because the cache index and queues are stored in shared memory accessible to any process with enough permissions). However, the above approach with a fake server is easier if you just want to cache a single file. Please note that I do not really understand all the complexities of the overall problem you are trying to solve (the description of the problem that you posted earlier was too convoluted for me) so I cannot recommend a good solution or evaluate whether the above is the right path towards that solution. I am just trying to answer a specific question instead. HTH, Alex. Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: adding content to the cache
On 08.05.2012 03:48, Alex Rousskov wrote: On 05/07/2012 05:35 AM, anita wrote: It sends an HTTP PUT request to Squid and reads the response. The PUT request body is taken from the named file. You most likely do not need this because you want to GET content from the fake server, and not PUT content to the fake server. Squid does not cache request bodies. Just came across this post. Wondering how to use this feature for squidclient. Can we simply do a squidclient -P http://server/filename.html ? Where the filename.html is the actual object that I would need to be pushed into the squid cache? IIRC, "squidclient -P Foo" specifies that the PUT request body should come from a file named Foo. Squid does not cache request bodies. Squid caches responses. If you want Squid to cache the contents of file Foo, you have to make Foo available on some web server. It is relatively easy to set up a "fake" server that responds with Foo to all HTTP requests (and runs on the same host as Squid), but you can also use Apache httpd or another full-blown origin server. With Rock Store (v3.2 and up), it is also possible to add something to the Squid disk cache without using HTTP (because the cache index and queues are stored in shared memory accessible to any process with enough permissions). However, the above approach with a fake server is easier if you just want to cache a single file. Please note that I do not really understand all the complexities of the overall problem you are trying to solve (the description of the problem that you posted earlier was too convoluted for me) so I cannot recommend a good solution or evaluate whether the above is the right path towards that solution. I am just trying to answer a specific question instead. HTH, Alex. Further to this. The HTTP specs do make PUT request bodies cacheable under the URL the request specifies. Squid just does not support that edge case of HTTP/1.1 yet. It would be nice to get it implemented for the next Squid series (3.3+) but its not clear yet whether this is the best solution for you. Consider where that Foo file comes from? how did it get to the updating machine in the first place? and why does that machine not simply pass squid a GET request for the URL to pre-cache the request for later use (no need to know what the content is in advance of the request, just GET with a max-age=0 control header)? Amos
Re: adding content to the cache
On 05/07/2012 05:35 AM, anita wrote: >> It sends an HTTP PUT request to Squid and reads the response. The PUT >> request body is taken from the named file. You most likely do not need >> this because you want to GET content from the fake server, and not PUT >> content to the fake server. Squid does not cache request bodies. > Just came across this post. Wondering how to use this feature for > squidclient. > Can we simply do a squidclient -P http://server/filename.html ? > Where the filename.html is the actual object that I would need to be pushed > into the squid cache? IIRC, "squidclient -P Foo" specifies that the PUT request body should come from a file named Foo. Squid does not cache request bodies. Squid caches responses. If you want Squid to cache the contents of file Foo, you have to make Foo available on some web server. It is relatively easy to set up a "fake" server that responds with Foo to all HTTP requests (and runs on the same host as Squid), but you can also use Apache httpd or another full-blown origin server. With Rock Store (v3.2 and up), it is also possible to add something to the Squid disk cache without using HTTP (because the cache index and queues are stored in shared memory accessible to any process with enough permissions). However, the above approach with a fake server is easier if you just want to cache a single file. Please note that I do not really understand all the complexities of the overall problem you are trying to solve (the description of the problem that you posted earlier was too convoluted for me) so I cannot recommend a good solution or evaluate whether the above is the right path towards that solution. I am just trying to answer a specific question instead. HTH, Alex.
Re: adding content to the cache
>It sends an HTTP PUT request to Squid and reads the response. The PUT >request body is taken from the named file. You most likely do not need >this because you want to GET content from the fake server, and not PUT >content to the fake server. Squid does not cache request bodies. Hi Alex, Just came across this post. Wondering how to use this feature for squidclient. Can we simply do a squidclient -P http://server/filename.html ? Where the filename.html is the actual object that I would need to be pushed into the squid cache? Or should it be embedded with some other data for squid to process the PUT request? I am working on squid 3.1.16. Thanks. -Anita -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/adding-content-to-the-cache-tp1048187p4614574.html Sent from the Squid - Development mailing list archive at Nabble.com.
Re: adding content to the cache -- a revisit
mån 2009-10-12 klockan 07:57 -0600 skrev bergenp...@comcast.net: > The content I'm trying to manually install into the squid server will be > a subset of the origin server content, so for objects not manually > installed into squid, squid will still need to go directly back to the > origin server. What you need is a) A HTTP server on the Squid server capable of serving the objects using HTTP, preverably with as identical properties as possible as the origin. This includes at least properties such as ETag, Content-Type, Content-Language, Content-Encoding and Last-Modified. b) wget, squidclient or another simple HTTP client capable of requesting URLs from the proxy. c) cache_peer line telling Squid that this local http server exists d) A unique http_port bound on the loopback interface, only used for this purpose (simplifies next step) e) cache_peer_access + never_direct rules telling Squid to fetch content requested from the unique port defined in 'd' from the peer defined in 'c' and only then.. Regards Henrik
adding content to the cache -- a revisit
There was a post on this same subject earlier this year. Want to post again to see if the solution space might have changed and if not, to get clarity on the suggested solutions. I've got a subset of content from a origin server that I would like to manually install into squid. The content pages are fixed content wise but I need to install new pages several times a minute. Much like the poster back in May, I was hoping there was some simple squid command that I could run to tell squid: 1) the web content page file name (which would already exist on the squid server) and 2) the name clients would use to http get this web page (and/or more complete object treatment info). Since the content will already exist on the squid server, I want to avoid the approach of having squid pull the content from the origin server. The content I'm trying to manually install into the squid server will be a subset of the origin server content, so for objects not manually installed into squid, squid will still need to go directly back to the origin server. Thanks
Re: adding content to the cache
On 05/12/2009 01:52 PM, Laurent Luce wrote: > Is there an easy way to check if a specific URL has been cached by > Squid ? I know about doing a head request to Squid but is there > another way to do that ? HEAD request with an only-if-cached Cache-Control directive is probably the best way to do it. Not sure whether Squid will honor only-if-cached for HEAD requests, but it should. Please test. Wget and curl can do that check. I did not check squidclient, but I suspect it can do it too. HTH, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Tuesday, May 12, 2009 12:33:35 PM > Subject: Re: adding content to the cache > > On 05/12/2009 11:03 AM, Laurent Luce wrote: > >> I think we are going to use a fake server and ask Squid to go get the >> file. >> >> Can you describe what squidclient -P (put file) does exactly. > > It sends an HTTP PUT request to Squid and reads the response. The PUT > request body is taken from the named file. You most likely do not need > this because you want to GET content from the fake server, and not PUT > content to the fake server. Squid does not cache request bodies. > >> Are you planning to do a segment cache in the future. This would be >> interesting especially for video caching. Or maybe it is there >> already and I missed it. Let me know. > > What is a "segment cache"? Caching of partial responses? Caching one > response across several cache_dirs? > > Thanks, > > Alex. > > >> - Original Message >> From: Alex Rousskov >> To: Laurent Luce >> Sent: Sunday, May 10, 2009 9:36:27 PM >> Subject: Re: adding content to the cache >> >> On 05/10/2009 08:57 PM, Laurent Luce wrote: >> >>> We thought of using a web server and getting the object using Squid. >>> We were hoping for a way of telling Squid that an object has been >>> added to its cache using some API. Reading your email, it seems to be >>> quite complicated. We would have preferred to keep the download part >>> separated from Squid and this is kind of a requirement for us. I >>> think we will explore the server solution and ask Squid to request >>> the object itself. >> It is difficult for me to get into details without knowing more about >> your environment and requirements. If you cannot use the existing >> "external" cache push APIs via a simple fake "server" or "peer" hack, >> eCAP may be the second overall best option that would still not require >> duplicating existing code and maintaining a lot of custom code. >> >> Good luck, >> >> Alex. >> >> >>> - Original Message >>> From: Alex Rousskov >>> To: Laurent Luce >>> Cc: squid-dev@squid-cache.org >>> Sent: Sunday, May 10, 2009 2:22:16 PM >>> Subject: Re: adding content to the cache >>> >>> On 05/10/2009 02:08 PM, Laurent Luce wrote: >>> >>>> I am looking into adding objects at runtime, on a regular basis, >>>> probably 2 or 3 every minute or so. Those objects are stored on the >>>> machine running Squid and I also have the HTTP headers for those >>>> objects. I just want to support one storage scheme (the one by >>>> default I guess). >>>> >>>> Looking at your answer, you are saying that adding objects at runtime >>>> is quite complicated due to the way Squid keeps that information >>>> internally, am I correct ? >>> Squid keeps some store information in RAM. If you update storage >>> externally, you will need some kind of communication channel to update >>> that information if you want Squid to notice the changes in store >>> information. You will also need to take care of conflicts. AFAICT, >>> implementing that would be a waste of time for your use case. >>> >>> You could implement store updates from the running Squid process itself. >>> This will work and will be very efficient. You may be able to use eCAP >>> request satisfaction, so that you do not have to learn a lot of Squid >>> code, deal with its complexities, and keep your custom patch in sync >>> with Squid code changes. AFAICT, this is _not_ the best way forward, but >>> I may be missing some important requirement. Please let me know if I am. >>> >>> >>>> I started looking at store*.c files to see if I could modify the >>>> store at runtime and add those objects. Any approach would you >>>> recommend if I wanted to go down
Re: adding content to the cache
Ok. I was hoping that squidclient could add content to the cache using the -P flag but I was wrong. Is there an easy way to check if a specific URL has been cached by Squid ? I know about doing a head request to Squid but is there another way to do that ? A segment-based cache is able to cache partial objects like the first 100 bytes of a video for example. This improve efficiency for large objects. There is a white paper on a solution called SProxy using that concept. I believe that Squid only caches full objects, correct ? Laurent - Original Message From: Alex Rousskov To: Laurent Luce Cc: squid-dev@squid-cache.org Sent: Tuesday, May 12, 2009 12:33:35 PM Subject: Re: adding content to the cache On 05/12/2009 11:03 AM, Laurent Luce wrote: > I think we are going to use a fake server and ask Squid to go get the > file. > > Can you describe what squidclient -P (put file) does exactly. It sends an HTTP PUT request to Squid and reads the response. The PUT request body is taken from the named file. You most likely do not need this because you want to GET content from the fake server, and not PUT content to the fake server. Squid does not cache request bodies. > Are you planning to do a segment cache in the future. This would be > interesting especially for video caching. Or maybe it is there > already and I missed it. Let me know. What is a "segment cache"? Caching of partial responses? Caching one response across several cache_dirs? Thanks, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Sent: Sunday, May 10, 2009 9:36:27 PM > Subject: Re: adding content to the cache > > On 05/10/2009 08:57 PM, Laurent Luce wrote: > >> We thought of using a web server and getting the object using Squid. >> We were hoping for a way of telling Squid that an object has been >> added to its cache using some API. Reading your email, it seems to be >> quite complicated. We would have preferred to keep the download part >> separated from Squid and this is kind of a requirement for us. I >> think we will explore the server solution and ask Squid to request >> the object itself. > > It is difficult for me to get into details without knowing more about > your environment and requirements. If you cannot use the existing > "external" cache push APIs via a simple fake "server" or "peer" hack, > eCAP may be the second overall best option that would still not require > duplicating existing code and maintaining a lot of custom code. > > Good luck, > > Alex. > > >> - Original Message >> From: Alex Rousskov >> To: Laurent Luce >> Cc: squid-dev@squid-cache.org >> Sent: Sunday, May 10, 2009 2:22:16 PM >> Subject: Re: adding content to the cache >> >> On 05/10/2009 02:08 PM, Laurent Luce wrote: >> >>> I am looking into adding objects at runtime, on a regular basis, >>> probably 2 or 3 every minute or so. Those objects are stored on the >>> machine running Squid and I also have the HTTP headers for those >>> objects. I just want to support one storage scheme (the one by >>> default I guess). >>> >>> Looking at your answer, you are saying that adding objects at runtime >>> is quite complicated due to the way Squid keeps that information >>> internally, am I correct ? >> Squid keeps some store information in RAM. If you update storage >> externally, you will need some kind of communication channel to update >> that information if you want Squid to notice the changes in store >> information. You will also need to take care of conflicts. AFAICT, >> implementing that would be a waste of time for your use case. >> >> You could implement store updates from the running Squid process itself. >> This will work and will be very efficient. You may be able to use eCAP >> request satisfaction, so that you do not have to learn a lot of Squid >> code, deal with its complexities, and keep your custom patch in sync >> with Squid code changes. AFAICT, this is _not_ the best way forward, but >> I may be missing some important requirement. Please let me know if I am. >> >> >>> I started looking at store*.c files to see if I could modify the >>> store at runtime and add those objects. Any approach would you >>> recommend if I wanted to go down this road. >> I would recommend reusing the existing code: >> >> 1) When an object needs to be added, request it from Squid using an >> off-the-shelf HTTP client like wget or curl. The client will talk to >> Squid using HTTP. >> >> 2) Write a very simple script that would serve the right object with the
Re: adding content to the cache
On 05/12/2009 11:03 AM, Laurent Luce wrote: > I think we are going to use a fake server and ask Squid to go get the > file. > > Can you describe what squidclient -P (put file) does exactly. It sends an HTTP PUT request to Squid and reads the response. The PUT request body is taken from the named file. You most likely do not need this because you want to GET content from the fake server, and not PUT content to the fake server. Squid does not cache request bodies. > Are you planning to do a segment cache in the future. This would be > interesting especially for video caching. Or maybe it is there > already and I missed it. Let me know. What is a "segment cache"? Caching of partial responses? Caching one response across several cache_dirs? Thanks, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Sent: Sunday, May 10, 2009 9:36:27 PM > Subject: Re: adding content to the cache > > On 05/10/2009 08:57 PM, Laurent Luce wrote: > >> We thought of using a web server and getting the object using Squid. >> We were hoping for a way of telling Squid that an object has been >> added to its cache using some API. Reading your email, it seems to be >> quite complicated. We would have preferred to keep the download part >> separated from Squid and this is kind of a requirement for us. I >> think we will explore the server solution and ask Squid to request >> the object itself. > > It is difficult for me to get into details without knowing more about > your environment and requirements. If you cannot use the existing > "external" cache push APIs via a simple fake "server" or "peer" hack, > eCAP may be the second overall best option that would still not require > duplicating existing code and maintaining a lot of custom code. > > Good luck, > > Alex. > > >> - Original Message >> From: Alex Rousskov >> To: Laurent Luce >> Cc: squid-dev@squid-cache.org >> Sent: Sunday, May 10, 2009 2:22:16 PM >> Subject: Re: adding content to the cache >> >> On 05/10/2009 02:08 PM, Laurent Luce wrote: >> >>> I am looking into adding objects at runtime, on a regular basis, >>> probably 2 or 3 every minute or so. Those objects are stored on the >>> machine running Squid and I also have the HTTP headers for those >>> objects. I just want to support one storage scheme (the one by >>> default I guess). >>> >>> Looking at your answer, you are saying that adding objects at runtime >>> is quite complicated due to the way Squid keeps that information >>> internally, am I correct ? >> Squid keeps some store information in RAM. If you update storage >> externally, you will need some kind of communication channel to update >> that information if you want Squid to notice the changes in store >> information. You will also need to take care of conflicts. AFAICT, >> implementing that would be a waste of time for your use case. >> >> You could implement store updates from the running Squid process itself. >> This will work and will be very efficient. You may be able to use eCAP >> request satisfaction, so that you do not have to learn a lot of Squid >> code, deal with its complexities, and keep your custom patch in sync >> with Squid code changes. AFAICT, this is _not_ the best way forward, but >> I may be missing some important requirement. Please let me know if I am. >> >> >>> I started looking at store*.c files to see if I could modify the >>> store at runtime and add those objects. Any approach would you >>> recommend if I wanted to go down this road. >> I would recommend reusing the existing code: >> >> 1) When an object needs to be added, request it from Squid using an >> off-the-shelf HTTP client like wget or curl. The client will talk to >> Squid using HTTP. >> >> 2) Write a very simple script that would serve the right object with the >> right headers, pretending to be an HTTP server or a peer cache. Any >> popular scripting language will have an HTTP library that will make >> writing such a fake server easy. Squid will talk to your script using HTTP. >> >> 3) If you are pushing objects for URLs that have different entities if >> accessed directly, then you will need to play with configuration so that >> only requests from your HTTP client are routed to your fake HTTP >> server/peer. I do not have a recipe, but a simple IP- or header-based >> ACL may be sufficient. >> >> Your 3/minute push rate is very low so no further optimizations are >> probably necessary. If your object
Re: adding content to the cache
I think we are going to use a fake server and ask Squid to go get the file. Can you describe what squidclient -P (put file) does exactly. Are you planning to do a segment cache in the future. This would be interesting especially for video caching. Or maybe it is there already and I missed it. Let me know. Laurent - Original Message From: Alex Rousskov To: Laurent Luce Sent: Sunday, May 10, 2009 9:36:27 PM Subject: Re: adding content to the cache On 05/10/2009 08:57 PM, Laurent Luce wrote: > We thought of using a web server and getting the object using Squid. > We were hoping for a way of telling Squid that an object has been > added to its cache using some API. Reading your email, it seems to be > quite complicated. We would have preferred to keep the download part > separated from Squid and this is kind of a requirement for us. I > think we will explore the server solution and ask Squid to request > the object itself. It is difficult for me to get into details without knowing more about your environment and requirements. If you cannot use the existing "external" cache push APIs via a simple fake "server" or "peer" hack, eCAP may be the second overall best option that would still not require duplicating existing code and maintaining a lot of custom code. Good luck, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Sunday, May 10, 2009 2:22:16 PM > Subject: Re: adding content to the cache > > On 05/10/2009 02:08 PM, Laurent Luce wrote: > >> I am looking into adding objects at runtime, on a regular basis, >> probably 2 or 3 every minute or so. Those objects are stored on the >> machine running Squid and I also have the HTTP headers for those >> objects. I just want to support one storage scheme (the one by >> default I guess). >> >> Looking at your answer, you are saying that adding objects at runtime >> is quite complicated due to the way Squid keeps that information >> internally, am I correct ? > > Squid keeps some store information in RAM. If you update storage > externally, you will need some kind of communication channel to update > that information if you want Squid to notice the changes in store > information. You will also need to take care of conflicts. AFAICT, > implementing that would be a waste of time for your use case. > > You could implement store updates from the running Squid process itself. > This will work and will be very efficient. You may be able to use eCAP > request satisfaction, so that you do not have to learn a lot of Squid > code, deal with its complexities, and keep your custom patch in sync > with Squid code changes. AFAICT, this is _not_ the best way forward, but > I may be missing some important requirement. Please let me know if I am. > > >> I started looking at store*.c files to see if I could modify the >> store at runtime and add those objects. Any approach would you >> recommend if I wanted to go down this road. > > I would recommend reusing the existing code: > > 1) When an object needs to be added, request it from Squid using an > off-the-shelf HTTP client like wget or curl. The client will talk to > Squid using HTTP. > > 2) Write a very simple script that would serve the right object with the > right headers, pretending to be an HTTP server or a peer cache. Any > popular scripting language will have an HTTP library that will make > writing such a fake server easy. Squid will talk to your script using HTTP. > > 3) If you are pushing objects for URLs that have different entities if > accessed directly, then you will need to play with configuration so that > only requests from your HTTP client are routed to your fake HTTP > server/peer. I do not have a recipe, but a simple IP- or header-based > ACL may be sufficient. > > Your 3/minute push rate is very low so no further optimizations are > probably necessary. If your objects are huge, you could optimize so that > the client does not have to receive the content even though Squid > fetches and stores everything. This would be similar to how some Range > and some IMS requests are processed by Squid. > > The above approach will work with any storage scheme. Do you see any > compelling reason to implement what you want inside Squid instead? > > Thank you, > > Alex. > > >> - Original Message >> From: Alex Rousskov >> To: Laurent Luce >> Cc: squid-dev@squid-cache.org >> Sent: Sunday, May 10, 2009 12:01:20 PM >> Subject: Re: adding content to the cache >> >> On 05/09/2009 08:04 PM, Laurent Luce wrote: >> >>> Actually, I am looking at a way of adding it directl
Re: adding content to the cache
Thanks Alex for those details, it is very helpful. We thought of using a web server and getting the object using Squid. We were hoping for a way of telling Squid that an object has been added to its cache using some API. Reading your email, it seems to be quite complicated. We would have preferred to keep the download part separated from Squid and this is kind of a requirement for us. I think we will explore the server solution and ask Squid to request the object itself. Laurent - Original Message From: Alex Rousskov To: Laurent Luce Cc: squid-dev@squid-cache.org Sent: Sunday, May 10, 2009 2:22:16 PM Subject: Re: adding content to the cache On 05/10/2009 02:08 PM, Laurent Luce wrote: > I am looking into adding objects at runtime, on a regular basis, > probably 2 or 3 every minute or so. Those objects are stored on the > machine running Squid and I also have the HTTP headers for those > objects. I just want to support one storage scheme (the one by > default I guess). > > Looking at your answer, you are saying that adding objects at runtime > is quite complicated due to the way Squid keeps that information > internally, am I correct ? Squid keeps some store information in RAM. If you update storage externally, you will need some kind of communication channel to update that information if you want Squid to notice the changes in store information. You will also need to take care of conflicts. AFAICT, implementing that would be a waste of time for your use case. You could implement store updates from the running Squid process itself. This will work and will be very efficient. You may be able to use eCAP request satisfaction, so that you do not have to learn a lot of Squid code, deal with its complexities, and keep your custom patch in sync with Squid code changes. AFAICT, this is _not_ the best way forward, but I may be missing some important requirement. Please let me know if I am. > I started looking at store*.c files to see if I could modify the > store at runtime and add those objects. Any approach would you > recommend if I wanted to go down this road. I would recommend reusing the existing code: 1) When an object needs to be added, request it from Squid using an off-the-shelf HTTP client like wget or curl. The client will talk to Squid using HTTP. 2) Write a very simple script that would serve the right object with the right headers, pretending to be an HTTP server or a peer cache. Any popular scripting language will have an HTTP library that will make writing such a fake server easy. Squid will talk to your script using HTTP. 3) If you are pushing objects for URLs that have different entities if accessed directly, then you will need to play with configuration so that only requests from your HTTP client are routed to your fake HTTP server/peer. I do not have a recipe, but a simple IP- or header-based ACL may be sufficient. Your 3/minute push rate is very low so no further optimizations are probably necessary. If your objects are huge, you could optimize so that the client does not have to receive the content even though Squid fetches and stores everything. This would be similar to how some Range and some IMS requests are processed by Squid. The above approach will work with any storage scheme. Do you see any compelling reason to implement what you want inside Squid instead? Thank you, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Sunday, May 10, 2009 12:01:20 PM > Subject: Re: adding content to the cache > > On 05/09/2009 08:04 PM, Laurent Luce wrote: > >> Actually, I am looking at a way of adding it directly to the squid >> cache. Basically, take the file and add it to the cache. I am looking >> into patching Squid to provide an API to do that. How complicated do >> you think it is if I want to add the file content along with the >> metadata directly into the cache ? > > Do you want to add objects runtime or offline? How many objects do you > need to add (ballpark estimates: hundreds, thousands, millions)? How > often do you need to add them (once, daily, weekly, etc.)? Are those > objects stored on the machine running Squid? How are those objects > stored now? Do stored objects come with HTTP response headers? > > If you want to add objects while Squid is running and want them to > become available as they are being addeded, fetching those objects using > wget/curl may be the best short-term solution (which can be optimized > and tuned using special headers and a local script pretending to be an > origin server). > > If you want to add objects offline, do you want to support multiple > Squid storage schemes (e.g., ufs, COSS, RockStore, etc.)? The best > implementation will probably depend on that and on the answers to the > questions abov
Re: adding content to the cache
On 05/10/2009 02:08 PM, Laurent Luce wrote: > I am looking into adding objects at runtime, on a regular basis, > probably 2 or 3 every minute or so. Those objects are stored on the > machine running Squid and I also have the HTTP headers for those > objects. I just want to support one storage scheme (the one by > default I guess). > > Looking at your answer, you are saying that adding objects at runtime > is quite complicated due to the way Squid keeps that information > internally, am I correct ? Squid keeps some store information in RAM. If you update storage externally, you will need some kind of communication channel to update that information if you want Squid to notice the changes in store information. You will also need to take care of conflicts. AFAICT, implementing that would be a waste of time for your use case. You could implement store updates from the running Squid process itself. This will work and will be very efficient. You may be able to use eCAP request satisfaction, so that you do not have to learn a lot of Squid code, deal with its complexities, and keep your custom patch in sync with Squid code changes. AFAICT, this is _not_ the best way forward, but I may be missing some important requirement. Please let me know if I am. > I started looking at store*.c files to see if I could modify the > store at runtime and add those objects. Any approach would you > recommend if I wanted to go down this road. I would recommend reusing the existing code: 1) When an object needs to be added, request it from Squid using an off-the-shelf HTTP client like wget or curl. The client will talk to Squid using HTTP. 2) Write a very simple script that would serve the right object with the right headers, pretending to be an HTTP server or a peer cache. Any popular scripting language will have an HTTP library that will make writing such a fake server easy. Squid will talk to your script using HTTP. 3) If you are pushing objects for URLs that have different entities if accessed directly, then you will need to play with configuration so that only requests from your HTTP client are routed to your fake HTTP server/peer. I do not have a recipe, but a simple IP- or header-based ACL may be sufficient. Your 3/minute push rate is very low so no further optimizations are probably necessary. If your objects are huge, you could optimize so that the client does not have to receive the content even though Squid fetches and stores everything. This would be similar to how some Range and some IMS requests are processed by Squid. The above approach will work with any storage scheme. Do you see any compelling reason to implement what you want inside Squid instead? Thank you, Alex. > - Original Message > From: Alex Rousskov > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Sunday, May 10, 2009 12:01:20 PM > Subject: Re: adding content to the cache > > On 05/09/2009 08:04 PM, Laurent Luce wrote: > >> Actually, I am looking at a way of adding it directly to the squid >> cache. Basically, take the file and add it to the cache. I am looking >> into patching Squid to provide an API to do that. How complicated do >> you think it is if I want to add the file content along with the >> metadata directly into the cache ? > > Do you want to add objects runtime or offline? How many objects do you > need to add (ballpark estimates: hundreds, thousands, millions)? How > often do you need to add them (once, daily, weekly, etc.)? Are those > objects stored on the machine running Squid? How are those objects > stored now? Do stored objects come with HTTP response headers? > > If you want to add objects while Squid is running and want them to > become available as they are being addeded, fetching those objects using > wget/curl may be the best short-term solution (which can be optimized > and tuned using special headers and a local script pretending to be an > origin server). > > If you want to add objects offline, do you want to support multiple > Squid storage schemes (e.g., ufs, COSS, RockStore, etc.)? The best > implementation will probably depend on that and on the answers to the > questions above. > > Thank you, > > Alex. > > >> - Original Message >> From: Amos Jeffries >> To: Laurent Luce >> Cc: squid-dev@squid-cache.org >> Sent: Friday, May 8, 2009 1:42:53 AM >> Subject: Re: adding content to the cache >> >> Laurent Luce wrote: >>> I am looking for a way to manually add content to the cache. Is there an >>> API to do that ? >>> >>> For >>> example, I have the following file image.gif and I want to add it to >>> the proxy cache so it can be served from there when needed. >>> >>> Laurent >>> >>
Re: adding content to the cache
I am looking into adding objects at runtime, on a regular basis, probably 2 or 3 every minute or so. Those objects are stored on the machine running Squid and I also have the HTTP headers for those objects. I just want to support one storage scheme (the one by default I guess). Looking at your answer, you are saying that adding objects at runtime is quite complicated due to the way Squid keeps that information internally, am I correct ? I started looking at store*.c files to see if I could modify the store at runtime and add those objects. Any approach would you recommend if I wanted to go down this road. Thanks. Laurent - Original Message From: Alex Rousskov To: Laurent Luce Cc: squid-dev@squid-cache.org Sent: Sunday, May 10, 2009 12:01:20 PM Subject: Re: adding content to the cache On 05/09/2009 08:04 PM, Laurent Luce wrote: > Actually, I am looking at a way of adding it directly to the squid > cache. Basically, take the file and add it to the cache. I am looking > into patching Squid to provide an API to do that. How complicated do > you think it is if I want to add the file content along with the > metadata directly into the cache ? Do you want to add objects runtime or offline? How many objects do you need to add (ballpark estimates: hundreds, thousands, millions)? How often do you need to add them (once, daily, weekly, etc.)? Are those objects stored on the machine running Squid? How are those objects stored now? Do stored objects come with HTTP response headers? If you want to add objects while Squid is running and want them to become available as they are being addeded, fetching those objects using wget/curl may be the best short-term solution (which can be optimized and tuned using special headers and a local script pretending to be an origin server). If you want to add objects offline, do you want to support multiple Squid storage schemes (e.g., ufs, COSS, RockStore, etc.)? The best implementation will probably depend on that and on the answers to the questions above. Thank you, Alex. > - Original Message > From: Amos Jeffries > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Friday, May 8, 2009 1:42:53 AM > Subject: Re: adding content to the cache > > Laurent Luce wrote: >> I am looking for a way to manually add content to the cache. Is there an API >> to do that ? >> >> For >> example, I have the following file image.gif and I want to add it to >> the proxy cache so it can be served from there when needed. >> >> Laurent >> > > Not at present. > You need to place it somewhere on a web server, assign it sufficient > cache-control settings to store for a long while and then request it from > squid. > > Common tools such as wget or squidclient can do that for you. > > Amos > -- Please be using > Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 > Current Beta Squid 3.1.0.7
Re: adding content to the cache
On 05/09/2009 08:04 PM, Laurent Luce wrote: > Actually, I am looking at a way of adding it directly to the squid > cache. Basically, take the file and add it to the cache. I am looking > into patching Squid to provide an API to do that. How complicated do > you think it is if I want to add the file content along with the > metadata directly into the cache ? Do you want to add objects runtime or offline? How many objects do you need to add (ballpark estimates: hundreds, thousands, millions)? How often do you need to add them (once, daily, weekly, etc.)? Are those objects stored on the machine running Squid? How are those objects stored now? Do stored objects come with HTTP response headers? If you want to add objects while Squid is running and want them to become available as they are being addeded, fetching those objects using wget/curl may be the best short-term solution (which can be optimized and tuned using special headers and a local script pretending to be an origin server). If you want to add objects offline, do you want to support multiple Squid storage schemes (e.g., ufs, COSS, RockStore, etc.)? The best implementation will probably depend on that and on the answers to the questions above. Thank you, Alex. > - Original Message > From: Amos Jeffries > To: Laurent Luce > Cc: squid-dev@squid-cache.org > Sent: Friday, May 8, 2009 1:42:53 AM > Subject: Re: adding content to the cache > > Laurent Luce wrote: >> I am looking for a way to manually add content to the cache. Is there an API >> to do that ? >> >> For >> example, I have the following file image.gif and I want to add it to >> the proxy cache so it can be served from there when needed. >> >> Laurent >> > > Not at present. > You need to place it somewhere on a web server, assign it sufficient > cache-control settings to store for a long while and then request it from > squid. > > Common tools such as wget or squidclient can do that for you. > > Amos > -- Please be using > Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 > Current Beta Squid 3.1.0.7
Re: adding content to the cache
Laurent Luce wrote: Actually, I am looking at a way of adding it directly to the squid cache. Basically, take the file and add it to the cache. I am looking into patching Squid to provide an API to do that. How complicated do you think it is if I want to add the file content along with the metadata directly into the cache ? You mean like implementing a POST to the cache_object:// data space? Adding stuff to the cache directories alone is fairly trivial. The problem lies with the internal indexing structures of Squid, which retain a mirror of the meta data plus additional data for memory management. If they are not updated Squid is likely to overwrite the new content at any time. Alex is looking into storage improvement stuff at present (RockStore feature in the wiki). I think getting in touch with him would be a benefit to both of you and see about collaborating work. Amos - Original Message From: Amos Jeffries To: Laurent Luce Cc: squid-dev@squid-cache.org Sent: Friday, May 8, 2009 1:42:53 AM Subject: Re: adding content to the cache Laurent Luce wrote: I am looking for a way to manually add content to the cache. Is there an API to do that ? For example, I have the following file image.gif and I want to add it to the proxy cache so it can be served from there when needed. Laurent Not at present. You need to place it somewhere on a web server, assign it sufficient cache-control settings to store for a long while and then request it from squid. Common tools such as wget or squidclient can do that for you. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.7 -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.7
Re: adding content to the cache
Actually, I am looking at a way of adding it directly to the squid cache. Basically, take the file and add it to the cache. I am looking into patching Squid to provide an API to do that. How complicated do you think it is if I want to add the file content along with the metadata directly into the cache ? Laurent - Original Message From: Amos Jeffries To: Laurent Luce Cc: squid-dev@squid-cache.org Sent: Friday, May 8, 2009 1:42:53 AM Subject: Re: adding content to the cache Laurent Luce wrote: > I am looking for a way to manually add content to the cache. Is there an API > to do that ? > > For > example, I have the following file image.gif and I want to add it to > the proxy cache so it can be served from there when needed. > > Laurent > Not at present. You need to place it somewhere on a web server, assign it sufficient cache-control settings to store for a long while and then request it from squid. Common tools such as wget or squidclient can do that for you. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.7
Re: adding content to the cache
Laurent Luce wrote: I am looking for a way to manually add content to the cache. Is there an API to do that ? For example, I have the following file image.gif and I want to add it to the proxy cache so it can be served from there when needed. Laurent Not at present. You need to place it somewhere on a web server, assign it sufficient cache-control settings to store for a long while and then request it from squid. Common tools such as wget or squidclient can do that for you. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.7
adding content to the cache
I am looking for a way to manually add content to the cache. Is there an API to do that ? For example, I have the following file image.gif and I want to add it to the proxy cache so it can be served from there when needed. Laurent
