On Apr 19, 2024, at 21:42, dave seddon <dave.seddon...@gmail.com> wrote:
Thanks Brian! Very cool
I don't know if you're supposed to retroactively update release notes, but that's a pretty big feature not in the notes.
Yeh the release notes for 10.x are not done yet. Stale-while-revaldate feature was a major challenge for some of our use cases, so it's awesome to hear that is resolved. Congratulations and thank you!
A potentially larger challenge is memory footprint. Generally, for Yahoo, Apple, Comcast CDNs, and all you guys all have beefy machines with plenty of RAM, so this isn't a drama. In those scenarios ATS is an amazing solution! Woot woot. Love it!
There are more unique situations, home caching for example, where you want large amounts of cache storage, but much lower in memory indexes to that storage.
I'm not suggesting ATS needs to solve is, but, for the sake of the mailing list history, people need to keep in mind.
Specifically, and unless I'm mistaken, RAM consumption is directly linear with the available cache size, regardless of the "working set".
E.g. If you have a large storage you need a (relatively) large amount of RAM to index into that storage, even if most of the storage is not accessed.
Well, yes. However it’s 10x less memory than e.g. squid for the same disk cache size :). The cost is 10 bytes of memory per directory entry (index).
Now, you can, and should, tune the number of directory entries to your data set. The default is pretty conservative, one dir entry for every 8KB of disk cache. If your average object size is noticeably larger than this (very likely), you can reduce memory usage accordingly. So setting this configuration to 16KB would reduce memory use in half, 32KB is 1/4 memory etc.
Cheers,
— Leif
Again, this is a totally understandable engineering decision, and makes sense in most situations. If I understand correctly, ATS does NOT support stale while revalidate, which is surprising given the RFC was created by a Yahoo-ligan. ;)
I'm glad you mentioned this. Actually Yahoo open-sourced the functionality for stale while revalidate via the stale_response.so plugin for ATS 10:
Depending on your use case, you also want to carefully consider the cache control headers for the downloaded object.
If I understand correctly, ATS does NOT support stale while revalidate, which is surprising given the RFC was created by a Yahoo-ligan. ;)
We discussed this in the slack channel. Probably the best option for you (other than writing a new plugin) is to use the background_fetch plugin. You will still need to trigger a download, but you can avoid having to wait for the full response in the client. So, what you’d do is
And make sure background_fetch is enabled on this remap rule. No matter what you do, you will have to have something that tells ATS to kick off a download, and the above is probably as good as any alternative.
Cheers,
— Leif
Hi Brian,
Thank you for your response. It'll do the job but, as far as I understand it, an external functionality will need to download the content in order to push it into the ATS. The content that I want to write in the ATS storage is on the Internet. Regards, Pavel. Hi Pavel,
This isn't a direct answer to your question, but are you aware of the ATS HTTP PUSH feature? That allows you to push objects into the ATS cache without the typical caching of proxied response. Can that help you in this situation?
On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov < pa...@x3me.net> wrote: Hi there,
Is there a way to use the ATS as an HTTP client to download and store given content? I'm aware that I can achieve this with a local HTTP client (wget, curl) which uses the ATS as a forward proxy and then the ATS will store the content (which is the actual goal). I was wondering if there is a way without using an additional HTTP client so that I can skip the additional content moving between sockets. Thanks, Pavel.
--
"Come to Me, all who are weary and heavy-laden, and I will
give you rest. Take My yoke upon you and learn from Me, for I am gentle and humble in heart, and you will find rest for your souls. For My yoke is easy and My burden is light." ~ Matthew 11:28-30
--
Regards,
Dave Seddon +1 415 857 5102
--
"Come to Me, all who are weary and heavy-laden, and I will
give you rest. Take My yoke upon you and learn from Me, for I am gentle and humble in heart, and you will find rest for your souls. For My yoke is easy and My burden is light." ~ Matthew 11:28-30
|
