Hi Gancho,
Let me explain the CDN content challenge which i'm trying to deal since
last couple of months. I believe the Cachekey plugin in v6.2 can help.
URL:
http://fs37.mydomain.com/9546/46cfd241f1da4ae9812f512f7b36643c/vlc-2.2.2-win64.exe
URL:
http://fs35.mydomain.com/9546/56dafac241f1da4ae9812f512f756bccf/vlc-2.2.2-win64.exe
URL:
http://fs31.mydomain.com/9546/436fd241f1da4ae9812f512f7b55667c/vlc-2.2.2-win64.exe
Random string in the URL "/9546/46cfd241f1da4ae9812f512f7b36643c" and
origin server "fs37,fs31" keeps on changing due to this the same object
is cached every time a user tries to download the file
I want to save the object once and should be deliver to client; for this
i need to normalize the cachekey and eliminate the random strings from
the cachekey in this way i would be able to avoid cache object
duplication and increase cache hit ratio (please correct me if im
wrong).
e.g filehippo site has below sequence:
When I click download button there are two requests one 301 which
contains (Location header for the requested content) and second 200:
GET
/download/file/6853a2c840eaefd1d7da43d6f2c94863adc5f470927402e6518d70573a99114d/
HTTP/1.1
Host: filehippo.com
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: FHSession=mfzdaugt4nu11q3yfxfkjyox;
FH_PreferredCulture=l=en-US&e=3/30/2017 1:38:22 PM;
__utmt_UA-5815250-1=1; __qca=P0-1359511593-1459345103148;
__utma=144473122.1934842269.1459345103.1459345103.1459345103.1;
__utmb=144473122.3.10.1459345119355; __utmc=144473122;
__utmz=144473122.1459345103.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
__utmv=144473122.|1=AB%20Test=new-home-v1=1
Referer:
http://filehippo.com/download_vlc_64/download/56a450f832aee6bb4fda3b01259f9866/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
HTTP/1.1 301 Moved Permanently
Accept-Ranges: bytes
Age: 0
Cache-Control: private
Connection: keep-alive
Content-Length: 0
Content-Type: text/html
Date: Wed, 30 Mar 2016 13:38:45 GMT
Location:
http://fs37.filehippo.com/9546/46cfd241f1da4ae9812f512f7b36643c/vlc-2.2.2-win64.exe
Via: 1.1 varnish
X-Cache: MISS
X-Cache-Hits: 0
x-debug-output: FHSession=mfzdaugt4nu11q3yfxfkjyox;
FH_PreferredCulture=l=en-US&e=3/30/2017 1:38:22 PM;
__utmt_UA-5815250-1=1; __qca=P0-1359511593-1459345103148;
__utma=144473122.1934842269.1459345103.1459345103.1459345103.1;
__utmb=144473122.3.10.1459345119355; __utmc=144473122;
__utmz=144473122.1459345103.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
__utmv=144473122.|1=AB%20Test=new-home-v1=1
X-Served-By: cache-lhr6334-LHR
200 Header: Why ATS is not caching octet stream despite having CONFIG
proxy.config.http.cache.required_headers INT 1
GET /9546/46cfd241f1da4ae9812f512f7b36643c/vlc-2.2.2-win64.exe HTTP/1.1
Host: fs37.filehippo.com
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: __utmt_UA-5815250-1=1; __qca=P0-1359511593-1459345103148;
__utma=144473122.1934842269.1459345103.1459345103.1459345103.1;
__utmb=144473122.3.10.1459345119355; __utmc=144473122;
__utmz=144473122.1459345103.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
__utmv=144473122.|1=AB%20Test=new-home-v1=1
Referer:
http://filehippo.com/download_vlc_64/download/56a450f832aee6bb4fda3b01259f9866/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 739
Connection: keep-alive
Content-Length: 31367109
Content-Type: application/octet-stream
Date: Wed, 30 Mar 2016 13:26:43 GMT
ETag: "81341be3a62d11:0"
Last-Modified: Mon, 02 Aug 2016 06:34:21 GMT
--
Regards,
Faisal.
------ Original Message ------
From: [email protected]
To: [email protected]
Sent: 8/2/2016 8:25:13 PM
Subject: Re[2]: Dynamic content Caching challenges
Hi
Im on way will catch you on irc today
--
Thanks
f.
Tuesday, 02 August 2016, 03:48AM +05:00 from Gancho Tenev
[email protected]:
Hi Faisal,
I am probably missing background/details since the configs you
provided did not make enough sense to me.
Maybe the following would work? (have not tested it since I don’t have
enough info)
regex_map http://(fs[0-9]+).filehippo.com
http://$1.filehippo.com@plugin=cachekey.so
@pparam=--static-prefix=filehippo.com
Could we just pick one use-case and work through it? For instance
“filehippo.com”.
Since cacheurl has been deprecated let us discuss cachekey configs.
Please provide:
- few samples of filehippo.com URIs
- the corresponding remap.config rule
- and then please describe how would you like URIs to match the
entries in the cache
so we can come up with the cachekey configs and we can take it from
there.
Cheers,
—Gancho
On Apr 13, 2016, at 11:22 AM, Muhammad Faisal <[email protected]>
wrote:
Hi,
I'm trying to deal with dynamic content to be cached by ATS. By
Dynamic I mean the URL for the actual content is always change this
results in wastage of Cache storage and low hit rate. As per my
understanding I have two challenges atm:
1- Websites with dynamic URL for requested content (e.g filehippo,
download.com etc etc)
2- Streaming web sites where the dynamic URL has 206 (partial
content)
I tried cacheurl plugin to assign as well as cachekey plugin. But i
couldn’t make the content cache friendly anyways. Below are the
configs i have tried so far:
cachekey Plugin configs are done on remap.config file as : regex_map
http://(fs[0-9]+).filehippo.com http://$1.filehippo.com
@plugin=cachekey.so
CacheURL plugin config:
http://.*[.]filehippo.com\/.*\/.*\/.*(\.exe)
http://cdn.filehippo.com/$1
http://.*\.gear3rd.net\/.*\/.*\/(.*\.mp4) http://cdn..gear3rd.net/$1
http://(cw[0-9]+).gear3rd.net\/\/files\/videos\/.*\/.*\/(.*\.mp4)
http://cdn.gear3rd.net/$1&$2
https?\:\/\/.*\/(.*\..*(mp4|3gp|flv))\?.*
http://video-file.ats.internal/$1
If someone has successfully configured the above scenario please help
me out as i dont have programming background to deal with this
complexity.
--
Regards,
Faisal.
