Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 13.05.16 21:25, Heiler Bemerguy пишет: > > > Ok but after enabling *collapsed_forwarding*, EVEN when disabling it again, the cache.log are full of *clientProcessHit: Vary object loop!* > Heh, I confirm. I see this when change Squid's binaries to 4.x when enabled collapsed_forwarding > > What happened ? My cache was modified ? collapsed is off now and even after restarting squid I'm still getting a flood of these messages which *were not there before*. > I suggest some cache contents changed. Also I suggest need to wipe out cache when change this setting. Or, another way, wait when content be replaced during time with native replacement policy... > > About the acl with "range_list_path", I've already done that to test.. it really alleviates the problem but it's still there. > > > Best Regards, > > -- > Heiler Bemerguy - (91) 98151-4894 > Assessor Técnico - CINBESA (91) 3184-1751 > > > Em 12/05/2016 20:21, joe escreveu: >> do not worry about vary its not a bug its the way its setup the vary >> handling yet it need plenty of work this is my guess after i look at the >> code >> >> for a range_offset_limit use this test im using it long time and its >> wonderful >> >> collapsed_forwarding on >> acl range_list_path urlpath_regex \.(mar|msp|esd|pkg\?) >> range_offset_limit -1 windows_list_path >> >> range_offset_limit 16 KB all !range_list_path#<---if you need this >> quick_abort_min 0 KB >> quick_abort_max 0 KB >> quick_abort_pct 100 >> >> im caching the above extension perfectly you can add to it >> >> >> >> >> -- >> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Getting-the-full-file-content-on-a-range-request-but-not-on-EVERY-get-tp4677503p4677543.html >> Sent from the Squid - Users mailing list archive at Nabble.com. >> ___ >> squid-users mailing list >> squid-users@lists.squid-cache.org >> http://lists.squid-cache.org/listinfo/squid-users > > > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNiIhAAoJENNXIZxhPexGus8IAKC2/iEobccG4Tkhs2zJb7Pd 1qFSDsnZRHHuz69cA29OrYKaH7qkvdtJo1k9becYbqdrcNLPprJgkUT/Qx/XIy0q NezsoGxMFDwJqbvS4tZLCI2u8/G8n2UvyjYsvUbzq7r6xViykOnOmKZLOh2z5m4H hwhBJsUJKyQxgHh3qk/ImHhmbM62vWOMID5cM2ibmjNLGKyWkGMA9VzIIVZl9TPw 4f6FuVy+CzA4NP7FuV85/Pqc91u2GGz6ZHojT9hL82hCR8iwhqdbwHA/w6gvP17i uZE5bq2Pl4IBw0fPbM/QneLHBMfIVBH5+apZJLM5DGKp2SAXc+g0rHRKZPT24cg= =rHna -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
Ok but after enabling *collapsed_forwarding*, EVEN when disabling it again, the cache.log are full of *clientProcessHit: Vary object loop!* What happened ? My cache was modified ? collapsed is off now and even after restarting squid I'm still getting a flood of these messages which *were not there before*. About the acl with "range_list_path", I've already done that to test.. it really alleviates the problem but it's still there. Best Regards, -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 Em 12/05/2016 20:21, joe escreveu: do not worry about vary its not a bug its the way its setup the vary handling yet it need plenty of work this is my guess after i look at the code for a range_offset_limit use this test im using it long time and its wonderful collapsed_forwarding on acl range_list_path urlpath_regex \.(mar|msp|esd|pkg\?) range_offset_limit -1 windows_list_path range_offset_limit 16 KB all !range_list_path#<---if you need this quick_abort_min 0 KB quick_abort_max 0 KB quick_abort_pct 100 im caching the above extension perfectly you can add to it -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Getting-the-full-file-content-on-a-range-request-but-not-on-EVERY-get-tp4677503p4677543.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Sat, 2016-05-14 at 01:52 +1200, Amos Jeffries wrote: > The default action should be to fetch each range request separately > and > in parallel. Not caching the results. > > When admin has set only the range offset & quick-abort to force full > object retrieval the behaviour Heiler mentions happens - lots of > upstream bandwidth used for N copies. > The first one to complete starts to be used as a HIT for future > reuqests. But as each of the initial transfers completes it replaces > the > previously cached object as the one being hit on. > > So timing is critical, if Squid happens to delay any of the parallel > requests just long enough in its TCP accept() queue, auth or ACL > processing they could become HITs on an already finished object. > > Amos Yes, you was right! Timing is very critical. I've simulated two concurrent transfers (without 'collapsed_forwarding on' and with 'range_offset_limit none') using this code: --- #!/bin/bash export http_proxy="127.0.0.1:3128" curl --range $((1024 * 1024 * 1))-$((1024 * 1024 * 2)) http://mirror.co mnet.uz/centos/7/os/x86_64/images/efiboot.img > /dev/null & curl --range $((1024 * 1024 * 3))-$((1024 * 1024 * 4)) http://mirror.co mnet.uz/centos/7/os/x86_64/images/efiboot.img > /dev/null --- And got two MISS': 1463150025.340987 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror. comnet.uz/centos/7/os/x86_64/images/efiboot.img - HIER_DIRECT/91.196.76.102 application/octet-stream 1463150026.315 1963 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror. comnet.uz/centos/7/os/x86_64/images/efiboot.img - HIER_DIRECT/91.196.76.102 application/octet-stream Then, I purged the object repeated with 'collapsed_forwarding on' and got MISS and HIT: 1463150169.010370 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror. comnet.uz/centos/7/os/x86_64/images/efiboot.img - HIER_DIRECT/91.196.76.102 application/octet-stream 1463150169.476836 127.0.0.1 TCP_HIT/206 1048950 GET http://mirror.c omnet.uz/centos/7/os/x86_64/images/efiboot.img - HIER_NONE/- application/octet-stream Amos, thank you very much for the detailed explanation. collapsed_forwarding is really useful option for this situation. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On 14/05/2016 12:38 a.m., Garri Djavadyan wrote: > On Fri, 2016-05-13 at 08:36 +1200, Amos Jeffries wrote: >> Have you given collapsed_forwarding a try? Its supposed to prevent >> all >> the duplicate requests making all those extra upstream connections >> unti >> at least the first one has finished getting the object. > > Amos, I believe that the above quote describes default Squid's action, > which does not require collapsed_forwarding. The details of my > experiments can be found here http://bugs.squid-cache.org/show_bug.cgi? > id=4511#c0. Thanks. What you are describing i that report is more along the lines of what can happen with collapsed_forwarding, shared cache by two workers, or such features linking the two requests which are supposed to be differently delay pooled. Heiler's problem is almost the opposite - the transactions *not* being linked, so they use upstream bandwidth over N connections for N times the bandwidth usage despite any per-message controls. Not just one connection using more data than expected. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On 14/05/2016 12:38 a.m., Garri Djavadyan wrote: > On Fri, 2016-05-13 at 08:36 +1200, Amos Jeffries wrote: >> Have you given collapsed_forwarding a try? Its supposed to prevent >> all >> the duplicate requests making all those extra upstream connections >> unti >> at least the first one has finished getting the object. > > Amos, I believe that the above quote describes default Squid's action, > which does not require collapsed_forwarding. The details of my > experiments can be found here http://bugs.squid-cache.org/show_bug.cgi? > id=4511#c0. Thanks. > The default action should be to fetch each range request separately and in parallel. Not caching the results. When admin has set only the range offset & quick-abort to force full object retrieval the behaviour Heiler mentions happens - lots of upstream bandwidth used for N copies. The first one to complete starts to be used as a HIT for future reuqests. But as each of the initial transfers completes it replaces the previously cached object as the one being hit on. So timing is critical, if Squid happens to delay any of the parallel requests just long enough in its TCP accept() queue, auth or ACL processing they could become HITs on an already finished object. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Fri, 2016-05-13 at 08:36 +1200, Amos Jeffries wrote: > Have you given collapsed_forwarding a try? Its supposed to prevent > all > the duplicate requests making all those extra upstream connections > unti > at least the first one has finished getting the object. Amos, I believe that the above quote describes default Squid's action, which does not require collapsed_forwarding. The details of my experiments can be found here http://bugs.squid-cache.org/show_bug.cgi? id=4511#c0. Thanks. Garri ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Thu, 2016-05-12 at 14:02 -0300, Heiler Bemerguy wrote: > > Hi Garri, > That bug report is mine.. lol Hi Heiler, Yes, I know it. I just tried to answer to the following question. > > > Is there a smart way to allow squid to download it from the > > > beginning > > > to the end (to actually cache it), but only on the FIRST > > > request/get? > > > Even if it makes the user wait for the full download, or cancel > > > it > > > temporarily, or.. whatever!! Anything!! The config option 'range_offset_limit none' (or -1) forces exactly that behavior. It fetches whole object from the beginning to the end on first range request. The problems you have encountered are consequences of the bug 4469 you reported. I encountered the same problems using Rock store. UFS/AUFS stores are not affected by the bug. So, to fix the problem more quickly, the best way is to provide to developers more information. For example, full debug log (ALL,9) for isolated requests. Thanks. Garri ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
do not worry about vary its not a bug its the way its setup the vary handling yet it need plenty of work this is my guess after i look at the code for a range_offset_limit use this test im using it long time and its wonderful collapsed_forwarding on acl range_list_path urlpath_regex \.(mar|msp|esd|pkg\?) range_offset_limit -1 windows_list_path range_offset_limit 16 KB all !range_list_path#<---if you need this quick_abort_min 0 KB quick_abort_max 0 KB quick_abort_pct 100 im caching the above extension perfectly you can add to it -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Getting-the-full-file-content-on-a-range-request-but-not-on-EVERY-get-tp4677503p4677543.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
Hi guys I just enabled "collapsed_forwarding" and noticed a lot of "Vary object loop!" that wasn't there before.. 2016/05/12 19:17:22 kid3| varyEvaluateMatch: Oops. Not a Vary match on second attempt, 'http://ego.globo.com/paparazzo/noticia/2016/05/tulio-maravilha-fala-de-video-intimo-com-mulher-gente-ve-e-deleta.html' 'accept-encoding="gzip,%20deflate,%20sdch", user-agent="Mozilla%2F5.0%20(Windows%20NT%206.1)%20AppleWebKit%2F537.36%20(KHTML,%20like%20Gecko)%20Chrome%2F50.0.2661.102%20Safari%2F537.36"' 2016/05/12 19:17:22 kid3| clientProcessHit: Vary object loop! 2016/05/12 19:17:22 kid3| varyEvaluateMatch: Oops. Not a Vary match on second attempt, 'http://ego.globo.com/fonts/typography.css' 'accept-encoding="gzip,%20deflate,%20sdch", user-agent="Mozilla%2F5.0%20(Windows%20NT%206.1)%20AppleWebKit%2F537.36%20(KHTML,%20like%20Gecko)%20Chrome%2F50.0.2661.102%20Safari%2F537.36"' 2016/05/12 19:17:22 kid3| clientProcessHit: Vary object loop! 2016/05/12 19:17:22 kid3| varyEvaluateMatch: Oops. Not a Vary match on second attempt, 'http://ego.globo.com/dynamo/scripts/js/glb.recaptcha.js' 'accept-encoding="gzip,%20deflate,%20sdch", user-agent="Mozilla%2F5.0%20(Windows%20NT%206.1)%20AppleWebKit%2F537.36%20(KHTML,%20like%20Gecko)%20Chrome%2F50.0.2661.102%20Safari%2F537.36"' I don't know if it's helping with the segmented downloads (that this thread is about...) though. Best Regards, -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 Em 12/05/2016 18:21, Alex Rousskov escreveu: On 05/12/2016 02:36 PM, Amos Jeffries wrote: Have you given collapsed_forwarding a try? Its supposed to prevent all the duplicate requests making all those extra upstream connections unti at least the first one has finished getting the object. For the record, collapsed forwarding collapses requests _before_ there is any response [header], not after the "first" request got the object [body]. Once there is a response [header], the usual caching code path kicks in and no collapsing is needed. Cache hits on that "usual caching code path" read from a "public" cache entry. Normally, those public entries are created when Squid receives a response [header]. Collapsed forwarding creates that entry before Squids gets the response [header], and, hence, before Squid can know for sure whether the response is going to be cachable, with all the risks that entails. Please do not misinterpret my email as a recommendation to give (or not to give) collapsed forwarding a try. I have _not_ analyzed the problems discussed on this thread. I just wanted to correct the description above. Nothing more. HTH, Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On 05/12/2016 02:36 PM, Amos Jeffries wrote: > Have you given collapsed_forwarding a try? Its supposed to prevent all > the duplicate requests making all those extra upstream connections unti > at least the first one has finished getting the object. For the record, collapsed forwarding collapses requests _before_ there is any response [header], not after the "first" request got the object [body]. Once there is a response [header], the usual caching code path kicks in and no collapsing is needed. Cache hits on that "usual caching code path" read from a "public" cache entry. Normally, those public entries are created when Squid receives a response [header]. Collapsed forwarding creates that entry before Squids gets the response [header], and, hence, before Squid can know for sure whether the response is going to be cachable, with all the risks that entails. Please do not misinterpret my email as a recommendation to give (or not to give) collapsed forwarding a try. I have _not_ analyzed the problems discussed on this thread. I just wanted to correct the description above. Nothing more. HTH, Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Amos, you're a genius! I had forgotten completely about this setting, oh my Idiotti . 13.05.16 2:36, Amos Jeffries пишет: > On 13/05/2016 7:17 a.m., Heiler Bemerguy wrote: >> >> I also don't care too much about duplicated cached files.. but trying to >> cache "ranged" requests is topping my link and in the end it seems it's >> not caching anything lol >> >> EVEN if I only allow range_offset to some urls or file extensions >> > > Have you given collapsed_forwarding a try? Its supposed to prevent all > the duplicate requests making all those extra upstream connections unti > at least the first one has finished getting the object. Combined with > the range_offset_limit and qick_abort_max to make that first request be > a full-fetch it might be able to solve your issue. > > Amos > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNO68AAoJENNXIZxhPexGQzcH/iulc1g/5hxya2gcLe0+FQsV 6Wtfl/DlpMkhwpKxPMMZMtcafyv7Umbbl2H7ErfcVGCVmZAWOZ0TYmA5dAu8NQ3X d8dQXAoKycNq3ZgKjpDIju33M4yJnPBrBb9M7oz1fXhj3WKo+LCapxX5nb9wYBgn CyrFcQf1rRSto2ly14w+j8XLJhyoUeBFoclJjksXbSycQmB0jPQtG9PkL3KOJNEv lYyTqbu0FUBoiek4oI0uOC2E9CjRPMyRdAWxkV5jEyqlRSfFo6bByEF9rKmk9CxH J0FOjpTvq5Bhrr4UVPp0jTpcuRebbLt6IxIdghU+fx6Fv33eRQwmSvxzHix0i0Y= =30FJ -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On 13/05/2016 7:17 a.m., Heiler Bemerguy wrote: > > I also don't care too much about duplicated cached files.. but trying to > cache "ranged" requests is topping my link and in the end it seems it's > not caching anything lol > > EVEN if I only allow range_offset to some urls or file extensions > Have you given collapsed_forwarding a try? Its supposed to prevent all the duplicate requests making all those extra upstream connections unti at least the first one has finished getting the object. Combined with the range_offset_limit and qick_abort_max to make that first request be a full-fetch it might be able to solve your issue. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 If a hammer, all, of course, is the nails :) 13.05.16 2:08, Yuri Voinov пишет: > > In comparison, the cache of thousands of Linux distributions, regardless of the purpose, of course, a trifle :) > > 13.05.16 2:07, Yuri Voinov пишет: > > > > I recently expressed the idea of caching torrents using > SQUID. :) What's an idea! I'm still impressed! :) > > > > > 13.05.16 2:02, Yuri Voinov пишет: > > > > > > > > Updates, > > > > > > > > > > in conjunction with hundreds OS's and distros, > better to do > > > with > > > > > > separate dedicated update server. IMHO. > > > > > > > > > > > > > > 13.05.16 1:56, Hans-Peter Jansen пишет: > > > > > > > On Freitag, 13. Mai 2016 01:09:39 Yuri Voinov > wrote: > > > > > > >> I suggest it is very bad idea to > transform caching > > > proxy to linux > > > > > > >> distro's or something else archive. > > > > > > > > > > > Yuri, if I wanted an archive, I would mirror > all stuff > > > and use local > > > > > > repos. > > > > > > It was sarcasm. And yes - local mirror is the best > approach. > > > > > > > > > > > I went that route for a long time - it's a > lot of work > > > to keep up > > > > > > everywhere, > > > > > > > and generates an awful amount of traffic (and > I did it > > > the sanest way > > > > > > possible > > > > > > My condolences. > > > > > > > > > > > - with a custom script, that was using > rsync..) > > > > > > > > > > >> As Amos said, "Squid is a cache, not an > archive". > > > > > > > > > > > Yes, updating 20 similar machines makes a > significant > > > difference with the > > > > > > > squid as a deduplicated cache - with no > recurring work > > > at all. > > > > > > Agree. Partially. With Solaris I've does it one > JumpStart > > > network > > > > > > server. Of course, the same technology is rara > avis in > > > modern > > > > > > world :) > > > > > > > > > > I now wonder - to sharpen the pencil you too > millstone ask? > > > :) My > > > > > > condolences again. > > > > > > > > > > > > > > > Pete > > > > > > > > ___ > > > > > > > squid-users mailing list > > > > > > > squid-users@lists.squid-cache.org > > > > > > > > http://lists.squid-cache.org/listinfo/squid-users > > > > > > > > > > > > > > -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNOMQAAoJENNXIZxhPexGQiQIAMX1XwoIr6WVtoSTpG8Jf8Vy +8jdMQzGMvG7DUkd+PUXu/b28HqcKm5yyTh8LkBt1tPYfetv5VBIY21yN8G/48Rb U9C+yYeLzQVUxikd5muel9G19khJ7XXqzJHSBNtPzlyN8+ylNX9TFFhmJZu4s6sq el05x/xtmlNkrW4AIbYF4Zz5HVxOResJS7yCvAuQw7LB010Hwed1PYyFw2baKVRO BZkiG8E8kxTPT6j+KB/nFZC2l0qmblPa2Tk3sjhKkngJx5rSbKTomZI075KRQilv VJgdYNp8/hDlvGszpi0/PSV8onia0ekF8jYU/7fkTsYqRJJ1P9G5f4ezZcr4Bos= =MaNc -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 In comparison, the cache of thousands of Linux distributions, regardless of the purpose, of course, a trifle :) 13.05.16 2:07, Yuri Voinov пишет: > > I recently expressed the idea of caching torrents using SQUID. :) What's an idea! I'm still impressed! :) > > 13.05.16 2:02, Yuri Voinov пишет: > > > > Updates, > > > > > in conjunction with hundreds OS's and distros, better to do > with > > > separate dedicated update server. IMHO. > > > > > > > 13.05.16 1:56, Hans-Peter Jansen пишет: > > > > On Freitag, 13. Mai 2016 01:09:39 Yuri Voinov wrote: > > > >> I suggest it is very bad idea to transform caching > proxy to linux > > > >> distro's or something else archive. > > > > > > Yuri, if I wanted an archive, I would mirror all stuff > and use local > > > repos. > > > It was sarcasm. And yes - local mirror is the best approach. > > > > > > I went that route for a long time - it's a lot of work > to keep up > > > everywhere, > > > > and generates an awful amount of traffic (and I did it > the sanest way > > > possible > > > My condolences. > > > > > > - with a custom script, that was using rsync..) > > > > > >> As Amos said, "Squid is a cache, not an archive". > > > > > > Yes, updating 20 similar machines makes a significant > difference with the > > > > squid as a deduplicated cache - with no recurring work > at all. > > > Agree. Partially. With Solaris I've does it one JumpStart > network > > > server. Of course, the same technology is rara avis in > modern > > > world :) > > > > > I now wonder - to sharpen the pencil you too millstone ask? > :) My > > > condolences again. > > > > > > > > Pete > > > > ___ > > > > squid-users mailing list > > > > squid-users@lists.squid-cache.org > > > > http://lists.squid-cache.org/listinfo/squid-users > > > > > > -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNOLSAAoJENNXIZxhPexGrL8H/2CENrZvFX4fUaZeCM55p88z ROVWCMpsx4YsD1zZieOIvUwBsBZlY35iBQ2R0gxEY8vrnNZE5eAt1+BK2zSxX2fZ us/JZ4yWeSlDYZ+foztnSDcPhqikdkaNXM12KjP7usnybx/dN2KaHdq8nc1xbOeN A1NCT4mnxiUNeycFOFfAHASvh4rCSii6MMqCS1Pf8PMwdTcKk/UudEBooytGlA7K gN39HWu+MCd1uUX2Mt9KRfSYQVx7OMjwWzNTH5Rpjk7gBd9a2oQGJrHoZ62xn0Pb 7ZCbZOgTJwovqWycf3Yv+6hb2m4g3xhJ9jBoM3RJjC4oRMynZ+Ls/QyBDfeUD0Q= =7c66 -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I recently expressed the idea of caching torrents using SQUID. :) What's an idea! I'm still impressed! :) 13.05.16 2:02, Yuri Voinov пишет: > > Updates, > > in conjunction with hundreds OS's and distros, better to do with > separate dedicated update server. IMHO. > > > 13.05.16 1:56, Hans-Peter Jansen пишет: > > On Freitag, 13. Mai 2016 01:09:39 Yuri Voinov wrote: > >> I suggest it is very bad idea to transform caching proxy to linux > >> distro's or something else archive. > > > Yuri, if I wanted an archive, I would mirror all stuff and use local > repos. > It was sarcasm. And yes - local mirror is the best approach. > > > I went that route for a long time - it's a lot of work to keep up > everywhere, > > and generates an awful amount of traffic (and I did it the sanest way > possible > My condolences. > > > - with a custom script, that was using rsync..) > > >> As Amos said, "Squid is a cache, not an archive". > > > Yes, updating 20 similar machines makes a significant difference with the > > squid as a deduplicated cache - with no recurring work at all. > Agree. Partially. With Solaris I've does it one JumpStart network > server. Of course, the same technology is rara avis in modern > world :) > > I now wonder - to sharpen the pencil you too millstone ask? :) My > condolences again. > > > > Pete > > ___ > > squid-users mailing list > > squid-users@lists.squid-cache.org > > http://lists.squid-cache.org/listinfo/squid-users > > -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNOKJAAoJENNXIZxhPexGWxMH+QEBHNNidx9Ar5Ec52bGz/Mm qpvkklZ3Tq6QPv2fimh8G7vXQghs29MfWCi6Q2ZRtWDBTbwXOBHAmhnkVt+VrzOH G7w61dmgmv9MfiotQu2f0g+v3v/MFWKjwEHU602SI8hMS4908Zw4z8Z6oGDzTi44 AdkoUX2J1SRLlVTuniVZJRGALnazHGTPiYHdvf5VpLnkWs1UtnugbrxLHDDRuVKi KsyIJvNv0NfvXa2HBGk7pgM4RWuuSc/bK1fZ6NzF7IccfqbqCUXZTo55IhzjS0kV fZKUIPk8Hm3Z2GaWvXO/9qtZiw4jMxGFmIDZCUjVDykJZFgdJZdsc87n0Lzbubw= =T9vF -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Updates, in conjunction with hundreds OS's and distros, better to do with separate dedicated update server. IMHO. 13.05.16 1:56, Hans-Peter Jansen пишет: > On Freitag, 13. Mai 2016 01:09:39 Yuri Voinov wrote: >> I suggest it is very bad idea to transform caching proxy to linux >> distro's or something else archive. > > Yuri, if I wanted an archive, I would mirror all stuff and use local repos. It was sarcasm. And yes - local mirror is the best approach. > > I went that route for a long time - it's a lot of work to keep up everywhere, > and generates an awful amount of traffic (and I did it the sanest way possible My condolences. > > - with a custom script, that was using rsync..) > >> As Amos said, "Squid is a cache, not an archive". > > Yes, updating 20 similar machines makes a significant difference with the > squid as a deduplicated cache - with no recurring work at all. Agree. Partially. With Solaris I've does it one JumpStart network server. Of course, the same technology is rara avis in modern world :) I now wonder - to sharpen the pencil you too millstone ask? :) My condolences again. > > > Pete > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNOE/AAoJENNXIZxhPexGSMsIAItiHicVTsVSI1u1Dn5Bb0M4 RjsQB3eG7ISoJKTe2nBBAeaRfoqcJxlahJ1Yabk+FP25zl+hmp0E1Yba2VTZvDFX 8BKwsNKuGPRdOgI5t69XLfgdQT21hnHNsYtH08pTSvvQeOYE9UA488jqHZKC20y7 J+3MU7aJiDZuSwfMsX5M9g1Wz6gQtgjuF4CL7ur7ssvhs+BD8ZUctJ7AjJ9JU2bf Z4uIIzKYhkNWZklPpB/ZVRr9qDfhiOpbit4oeI2ELrgU7ro4EcggNY24Nxel6CGX WzU3NDsv2mm3I5u3mW6KkWHYd2k/5uqE+cAXZSdRBBz2WTj2YJoMePnyMQIUowE= =i8rv -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Freitag, 13. Mai 2016 01:09:39 Yuri Voinov wrote: > I suggest it is very bad idea to transform caching proxy to linux > distro's or something else archive. Yuri, if I wanted an archive, I would mirror all stuff and use local repos. I went that route for a long time - it's a lot of work to keep up everywhere, and generates an awful amount of traffic (and I did it the sanest way possible - with a custom script, that was using rsync..) > As Amos said, "Squid is a cache, not an archive". Yes, updating 20 similar machines makes a significant difference with the squid as a deduplicated cache - with no recurring work at all. Pete ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 And I did not promise a silver bullet :) This is just a small workaround, which does not work in all cases. :) 13.05.16 1:17, Heiler Bemerguy пишет: > > > I also don't care too much about duplicated cached files.. but trying to cache "ranged" requests is topping my link and in the end it seems it's not caching anything lol > > EVEN if I only allow range_offset to some urls or file extensions > > > Best Regards, > > > -- > Heiler Bemerguy - (91) 98151-4894 > Assessor Técnico - CINBESA (91) 3184-1751 > > Em 12/05/2016 16:09, Yuri Voinov escreveu: > I suggest it is very bad idea to transform caching proxy to linux > distro's or something else archive. > > As Amos said, "Squid is a cache, not an archive". > > > 13.05.16 0:57, Hans-Peter Jansen пишет: > >>> Hi Heiler, > >>> > >>> On Donnerstag, 12. Mai 2016 13:28:00 Heiler Bemerguy wrote: > Hi Pete, thanks for replying... let me see if I got it right.. > > Will I need to specify every url/domain I want it to act on ? I want > squid to do it for every range-request downloads that should/would be > cached (based on other rules, pattern_refreshs etc) > >>> Yup, that's right. At least, that's the common approach to deal with CDNs. > >>> I think, that disallowing range requests is too drastic to work fine > on the > >>> long run, but let us know, if you get to satisfactory solution this way. > >>> > It doesn't need to delay any downloads as long as it isn't a dupe of > what's already being downloaded. > >>> You can set to delay to zero of course. > >>> > >>> This is only one side of the issues with CDNs. The other, more > problematic > >>> side of it is, that many server with different URLs provide the same > files. > >>> Every new address will result in a new download of otherwise identical > >>> content. > >>> > >>> Here's an example of openSUSE: > >>> > >>> # > >>> # this file was generated by gen_openSUSE_dedups > >>> # from http://mirrors.opensuse.org/list/all.html > >>> # with timestamp Thu, 12 May 2016 05:30:18 +0200 > >>> # > >>> [openSUSE] > >>> match: > >>> # openSUSE Headquarter > >>> http\:\/\/[a-z0-9]+\.opensuse\.org\/(.*) > >>> # South Africa (za) > >>> http\:\/\/ftp\.up\.ac\.za\/mirrors\/opensuse\/opensuse\/(.*) > >>> # Bangladesh (bd) > >>> http\:\/\/mirror\.dhakacom\.com\/opensuse\/(.*) > >>> http\:\/\/mirrors\.ispros\.com\.bd\/opensuse\/(.*) > >>> # China (cn) > >>> http\:\/\/mirror\.bjtu\.edu\.cn\/opensuse\/(.*) > >>> http\:\/\/fundawang\.lcuc\.org\.cn\/opensuse\/(.*) > >>> http\:\/\/mirrors\.tuna\.tsinghua\.edu\.cn\/opensuse\/(.*) > >>> http\:\/\/mirrors\.skyshe\.cn\/opensuse\/(.*) > >>> http\:\/\/mirrors\.hust\.edu\.cn\/opensuse\/(.*) > >>> http\:\/\/c\.mirrors\.lanunion\.org\/opensuse\/(.*) > >>> http\:\/\/mirrors\.hustunique\.com\/opensuse\/(.*) > >>> http\:\/\/mirrors\.sohu\.com\/opensuse\/(.*) > >>> http\:\/\/mirrors\.ustc\.edu\.cn\/opensuse\/(.*) > >>> # Hong Kong (hk) > >>> http\:\/\/mirror\.rackspace\.hk\/openSUSE\/(.*) > >>> # Indonesia (id) > >>> http\:\/\/mirror\.linux\.or\.id\/linux\/opensuse\/(.*) > >>> http\:\/\/buaya\.klas\.or\.id\/opensuse\/(.*) > >>> http\:\/\/kartolo\.sby\.datautama\.net\.id\/openSUSE\/(.*) > >>> http\:\/\/opensuse\.idrepo\.or\.id\/opensuse\/(.*) > >>> http\:\/\/mirror\.unej\.ac\.id\/opensuse\/(.*) > >>> http\:\/\/download\.opensuse\.or\.id\/(.*) > >>> http\:\/\/repo\.ugm\.ac\.id\/opensuse\/(.*) > >>> http\:\/\/dl2\.foss\-id\.web\.id\/opensuse\/(.*) > >>> # Israel (il) > >>> http\:\/\/mirror\.isoc\.org\.il\/pub\/opensuse\/(.*) > >>> > >>> [...] -> this list contains about 180 entries > >>> > >>> replace: http://download.opensuse.org.%(intdomain)s/\1 > >>> # fetch all redirected objects explicitly > >>> fetch: true > >>> > >>> > >>> This is, how CDNs work, but it's a nightmare for caching proxies. > >>> In such scenarios squid_dedup comes to rescue. > >>> > >>> Cheers, > >>> Pete > >>> ___ > >>> squid-users mailing list > >>> squid-users@lists.squid-cache.org > >>> http://lists.squid-cache.org/listinfo/squid-users >> >> >> >> ___ >> squid-users mailing list >> squid-users@lists.squid-cache.org >> http://lists.squid-cache.org/listinfo/squid-users > > > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNNcvAAoJENNXIZxhPexGLGsH/jilasMtRp499QJS+G5O9lHv z1MtLbZnExrzVcmqb69jsbaZWjwnhvUF1Ng7fZbep5q6pky3DGcwxQu/EBVhD3+p tTliArKa45dmhbOm5a0ljJcq73hBtUrlS0UrGDV6CMRrXjHjSUzy6+BwwsI1mClp dtOos3NoSDlQmazkEDA6+f3iuYykjinmTsFlJRgQipluXnUUlmvbnpwZHqUhTA0R X2I6j3zdTDHGszlXkoFrKg+Vj0gOzeGfA5IPx7/vnruShlYSPWuvoVfvi4ZLYV2y
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I suggest it is very bad idea to transform caching proxy to linux distro's or something else archive. As Amos said, "Squid is a cache, not an archive". 13.05.16 0:57, Hans-Peter Jansen пишет: > Hi Heiler, > > On Donnerstag, 12. Mai 2016 13:28:00 Heiler Bemerguy wrote: >> Hi Pete, thanks for replying... let me see if I got it right.. >> >> Will I need to specify every url/domain I want it to act on ? I want >> squid to do it for every range-request downloads that should/would be >> cached (based on other rules, pattern_refreshs etc) > > Yup, that's right. At least, that's the common approach to deal with CDNs. > I think, that disallowing range requests is too drastic to work fine on the > long run, but let us know, if you get to satisfactory solution this way. > >> It doesn't need to delay any downloads as long as it isn't a dupe of >> what's already being downloaded. > > You can set to delay to zero of course. > > This is only one side of the issues with CDNs. The other, more problematic > side of it is, that many server with different URLs provide the same files. > Every new address will result in a new download of otherwise identical > content. > > Here's an example of openSUSE: > > # > # this file was generated by gen_openSUSE_dedups > # from http://mirrors.opensuse.org/list/all.html > # with timestamp Thu, 12 May 2016 05:30:18 +0200 > # > [openSUSE] > match: > # openSUSE Headquarter > http\:\/\/[a-z0-9]+\.opensuse\.org\/(.*) > # South Africa (za) > http\:\/\/ftp\.up\.ac\.za\/mirrors\/opensuse\/opensuse\/(.*) > # Bangladesh (bd) > http\:\/\/mirror\.dhakacom\.com\/opensuse\/(.*) > http\:\/\/mirrors\.ispros\.com\.bd\/opensuse\/(.*) > # China (cn) > http\:\/\/mirror\.bjtu\.edu\.cn\/opensuse\/(.*) > http\:\/\/fundawang\.lcuc\.org\.cn\/opensuse\/(.*) > http\:\/\/mirrors\.tuna\.tsinghua\.edu\.cn\/opensuse\/(.*) > http\:\/\/mirrors\.skyshe\.cn\/opensuse\/(.*) > http\:\/\/mirrors\.hust\.edu\.cn\/opensuse\/(.*) > http\:\/\/c\.mirrors\.lanunion\.org\/opensuse\/(.*) > http\:\/\/mirrors\.hustunique\.com\/opensuse\/(.*) > http\:\/\/mirrors\.sohu\.com\/opensuse\/(.*) > http\:\/\/mirrors\.ustc\.edu\.cn\/opensuse\/(.*) > # Hong Kong (hk) > http\:\/\/mirror\.rackspace\.hk\/openSUSE\/(.*) > # Indonesia (id) > http\:\/\/mirror\.linux\.or\.id\/linux\/opensuse\/(.*) > http\:\/\/buaya\.klas\.or\.id\/opensuse\/(.*) > http\:\/\/kartolo\.sby\.datautama\.net\.id\/openSUSE\/(.*) > http\:\/\/opensuse\.idrepo\.or\.id\/opensuse\/(.*) > http\:\/\/mirror\.unej\.ac\.id\/opensuse\/(.*) > http\:\/\/download\.opensuse\.or\.id\/(.*) > http\:\/\/repo\.ugm\.ac\.id\/opensuse\/(.*) > http\:\/\/dl2\.foss\-id\.web\.id\/opensuse\/(.*) > # Israel (il) > http\:\/\/mirror\.isoc\.org\.il\/pub\/opensuse\/(.*) > > [...] -> this list contains about 180 entries > > replace: http://download.opensuse.org.%(intdomain)s/\1 > # fetch all redirected objects explicitly > fetch: true > > > This is, how CDNs work, but it's a nightmare for caching proxies. > In such scenarios squid_dedup comes to rescue. > > Cheers, > Pete > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNNTzAAoJENNXIZxhPexG8XIIAKal+I1GMvTS9QDdJT6pxi7n IL/d33/YUelZJ9ok1bLAiI1DNOJR6xwK6OZ+LefPOrxH1Q14quGJ5m873065jE+H /1qhYs8rVVQ8qlLQyMI+aacEA9FV7j6OpWMteM+54SSjLlW4z0pJkw+vSsMwCnI5 Sy3qryieIImtmYnT1wbVM5Pop3lrLA/t1jza619ioxIxWa4M4bSO2EAR+Qj5HiUg BT8ki8t1GIO12RatjqDwSouU+yDMK85amUKZBjRFXhyOxi1Cg+5uleI4C2lUjqM2 f1n3KBC7mlF6snAT74kc+JWLsNd2ohlkmJB8tSIhkxvkgmaWDpCpwaGaUmtkuXg= =/fDD -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
Hi Heiler, On Donnerstag, 12. Mai 2016 13:28:00 Heiler Bemerguy wrote: > Hi Pete, thanks for replying... let me see if I got it right.. > > Will I need to specify every url/domain I want it to act on ? I want > squid to do it for every range-request downloads that should/would be > cached (based on other rules, pattern_refreshs etc) Yup, that's right. At least, that's the common approach to deal with CDNs. I think, that disallowing range requests is too drastic to work fine on the long run, but let us know, if you get to satisfactory solution this way. > It doesn't need to delay any downloads as long as it isn't a dupe of > what's already being downloaded. You can set to delay to zero of course. This is only one side of the issues with CDNs. The other, more problematic side of it is, that many server with different URLs provide the same files. Every new address will result in a new download of otherwise identical content. Here's an example of openSUSE: # # this file was generated by gen_openSUSE_dedups # from http://mirrors.opensuse.org/list/all.html # with timestamp Thu, 12 May 2016 05:30:18 +0200 # [openSUSE] match: # openSUSE Headquarter http\:\/\/[a-z0-9]+\.opensuse\.org\/(.*) # South Africa (za) http\:\/\/ftp\.up\.ac\.za\/mirrors\/opensuse\/opensuse\/(.*) # Bangladesh (bd) http\:\/\/mirror\.dhakacom\.com\/opensuse\/(.*) http\:\/\/mirrors\.ispros\.com\.bd\/opensuse\/(.*) # China (cn) http\:\/\/mirror\.bjtu\.edu\.cn\/opensuse\/(.*) http\:\/\/fundawang\.lcuc\.org\.cn\/opensuse\/(.*) http\:\/\/mirrors\.tuna\.tsinghua\.edu\.cn\/opensuse\/(.*) http\:\/\/mirrors\.skyshe\.cn\/opensuse\/(.*) http\:\/\/mirrors\.hust\.edu\.cn\/opensuse\/(.*) http\:\/\/c\.mirrors\.lanunion\.org\/opensuse\/(.*) http\:\/\/mirrors\.hustunique\.com\/opensuse\/(.*) http\:\/\/mirrors\.sohu\.com\/opensuse\/(.*) http\:\/\/mirrors\.ustc\.edu\.cn\/opensuse\/(.*) # Hong Kong (hk) http\:\/\/mirror\.rackspace\.hk\/openSUSE\/(.*) # Indonesia (id) http\:\/\/mirror\.linux\.or\.id\/linux\/opensuse\/(.*) http\:\/\/buaya\.klas\.or\.id\/opensuse\/(.*) http\:\/\/kartolo\.sby\.datautama\.net\.id\/openSUSE\/(.*) http\:\/\/opensuse\.idrepo\.or\.id\/opensuse\/(.*) http\:\/\/mirror\.unej\.ac\.id\/opensuse\/(.*) http\:\/\/download\.opensuse\.or\.id\/(.*) http\:\/\/repo\.ugm\.ac\.id\/opensuse\/(.*) http\:\/\/dl2\.foss\-id\.web\.id\/opensuse\/(.*) # Israel (il) http\:\/\/mirror\.isoc\.org\.il\/pub\/opensuse\/(.*) [...] -> this list contains about 180 entries replace: http://download.opensuse.org.%(intdomain)s/\1 # fetch all redirected objects explicitly fetch: true This is, how CDNs work, but it's a nightmare for caching proxies. In such scenarios squid_dedup comes to rescue. Cheers, Pete ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 IMO better to use range_offset_limit none !dont_cache_url all to improve selectivity between non-cached and cached URL's with ACL's 12.05.16 23:02, Heiler Bemerguy пишет: > > > Hi Garri, > > That bug report is mine.. lol > > But I couldn't keep testing it to confirm if the problem was about ABORTING downloads or just trying to download what's already being downloaded... > > When you use quick_abort_min -1, it seems to "fix" the caching issue itself, but it won't prevent the concurrent downloads, which sucks up the link.. > > I don't know if it won't happen with aufs/ufs, I'm using only rock store. > > > -- > Heiler Bemerguy - (91) 98151-4894 > Assessor Técnico - CINBESA (91) 3184-1751 > > Em 12/05/2016 01:01, Garri Djavadyan escreveu: >> On Wed, 2016-05-11 at 21:37 -0300, Heiler Bemerguy wrote: >>> Hey guys, >>> First take a look at the log: >>> root@proxy:/var/log/squid# tail -f access.log |grep http://download.c >>> dn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar >>> 1463011781.572 8776 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 >>> application/octet-stream >>> 1463011851.008 9347 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463011920.683 9645 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 >>> application/octet-stream >>> 1463012000.144 19154 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463012072.276 12121 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463012145.643 13358 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463012217.472 11772 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463012294.676 17148 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> 1463012370.131 15272 10.1.3.236 TCP_MISS/206 300520 GET http://downl >>> oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- >>> BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 >>> application/octet-stream >>> Now think: An user is just doing a segmented/ranged download, right? >>> Squid won't cache the file because it is a range-download, not a full >>> file download. >>> But I WANT squid to cache it. So I decide to use "range_offset_limit >>> -1", but then on every GET squid will re-download the file from the >>> beginning, opening LOTs of simultaneous connections and using too >>> much bandwidth, doing just the OPPOSITE it's meant to! >>> >>> Is there a smart way to allow squid to download it from the beginning >>> to the end (to actually cache it), but only on the FIRST request/get? >>> Even if it makes the user wait for the full download, or cancel it >>> temporarily, or.. whatever!! Anything!! >>> >>> Best Regards, >>> -- >>> Heiler Bemerguy - (91) 98151-4894 >>> Assessor Técnico - CINBESA (91) 3184-1751 >>> ___ >>> squid-users mailing list >>> squid-users@lists.squid-cache.org >>> http://lists.squid-cache.org/listinfo/squid-users >> Hi, I believe, you describe the bug http://bugs.squid-cache.org/show_bu >> g.cgi?id=4469 >> >> I tried to reproduce the problem and have found that the problem >> appears only with rock storage configurations. Can you try with >> ufs/aufs storage? >> ___ >> squid-users mailing list >> squid-users@lists.squid-cache.org >> http://lists.squid-cache.org/listinfo/squid-users > > > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXNLi/AAoJENNXIZxhPexG5aAH/juZyvly/aSIguez9dAKPbbb AHpPxoky36FYOjlPbqmXjdrMPs9qNGT+Ns7WDxsNZFM0Cfbh5UgBfQc64kFoV0k/
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
Hi Garri, That bug report is mine.. lol But I couldn't keep testing it to confirm if the problem was about ABORTING downloads or just trying to download what's already being downloaded... When you use quick_abort_min -1, it seems to "fix" the caching issue itself, but it won't prevent the concurrent downloads, which sucks up the link.. I don't know if it won't happen with aufs/ufs, I'm using only rock store. -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 Em 12/05/2016 01:01, Garri Djavadyan escreveu: On Wed, 2016-05-11 at 21:37 -0300, Heiler Bemerguy wrote: Hey guys, First take a look at the log: root@proxy:/var/log/squid# tail -f access.log |grep http://download.c dn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar 1463011781.572 8776 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 application/octet-stream 1463011851.008 9347 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463011920.683 9645 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 application/octet-stream 1463012000.144 19154 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463012072.276 12121 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463012145.643 13358 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463012217.472 11772 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463012294.676 17148 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream 1463012370.131 15272 10.1.3.236 TCP_MISS/206 300520 GET http://downl oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 application/octet-stream Now think: An user is just doing a segmented/ranged download, right? Squid won't cache the file because it is a range-download, not a full file download. But I WANT squid to cache it. So I decide to use "range_offset_limit -1", but then on every GET squid will re-download the file from the beginning, opening LOTs of simultaneous connections and using too much bandwidth, doing just the OPPOSITE it's meant to! Is there a smart way to allow squid to download it from the beginning to the end (to actually cache it), but only on the FIRST request/get? Even if it makes the user wait for the full download, or cancel it temporarily, or.. whatever!! Anything!! Best Regards, -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users Hi, I believe, you describe the bug http://bugs.squid-cache.org/show_bu g.cgi?id=4469 I tried to reproduce the problem and have found that the problem appears only with rock storage configurations. Can you try with ufs/aufs storage? ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
Hi Pete, thanks for replying... let me see if I got it right.. Will I need to specify every url/domain I want it to act on ? I want squid to do it for every range-request downloads that should/would be cached (based on other rules, pattern_refreshs etc) It doesn't need to delay any downloads as long as it isn't a dupe of what's already being downloaded. Best Regards, -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 Em 12/05/2016 11:06, Hans-Peter Jansen escreveu: On Mittwoch, 11. Mai 2016 21:37:17 Heiler Bemerguy wrote: Hey guys, First take a look at the log: root@proxy:/var/log/squid# tail -f access.log |grep http://download.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt-> BR/firefox-45.0.1.complete.mar 1463011781.572 8776 10.1.3.236 TCP_MISS/206 300520 GET [...] Now think: An user is just doing a segmented/ranged download, right? Squid won't cache the file because it is a range-download, not a full file download. But I WANT squid to cache it. So I decide to use "range_offset_limit -1", but then on every GET squid will re-download the file from the beginning, opening LOTs of simultaneous connections and using too much bandwidth, doing just the OPPOSITE it's meant to! Is there a smart way to allow squid to download it from the beginning to the end (to actually cache it), but only on the FIRST request/get? Even if it makes the user wait for the full download, or cancel it temporarily, or.. whatever!! Anything!! Well, this is exactly, what my squid_dedup helper was created for! See my announcement: Subject: [squid-users] New StoreID helper: squid_dedup Date: Mon, 09 May 2016 23:56:45 +0200 My openSUSE environment is fetching _all_ updates with byte-ranges from many servers. Therefor, I created squid_dedup. Your specific config could look like this: /etc/squid/dedup/mozilla.conf: [mozilla] match: http\:\/\/download\.cdn\.mozilla\.net/(.*) replace: http://download.cdn.mozilla.net.%(intdomain)s/\1 fetch: true The fetch parameter is unique among the other StoreID helper (AFAIK): it is fetching the object after a certain delay with a pool of fetcher threads. The idea is: after the first access for an object, wait a bit (global setting, default: 15 secs), and then fetch the whole thing once. It won't solve anything for the first client, but for all subsequent accesses. The fetcher avoids fetching anything more than once by checking the http headers. This is a pretty new project, but be assured, that the basic functions are working fine, and I will do my best to solve any upcoming issues. It is implemented with Python3 and prepared for supporting additional features easily, while keeping a good part of an eye on efficiency. Let me know, if you're going to try it. Pete ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Mittwoch, 11. Mai 2016 21:37:17 Heiler Bemerguy wrote: > Hey guys, > > First take a look at the log: > > root@proxy:/var/log/squid# tail -f access.log |grep > http://download.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt-> > BR/firefox-45.0.1.complete.mar 1463011781.572 8776 10.1.3.236 TCP_MISS/206 > 300520 GET [...] > Now think: An user is just doing a segmented/ranged download, right? > Squid won't cache the file because it is a range-download, not a full > file download. > But I WANT squid to cache it. So I decide to use "range_offset_limit > -1", but then on every GET squid will re-download the file from the > beginning, opening LOTs of simultaneous connections and using too much > bandwidth, doing just the OPPOSITE it's meant to! > > Is there a smart way to allow squid to download it from the beginning to > the end (to actually cache it), but only on the FIRST request/get? Even > if it makes the user wait for the full download, or cancel it > temporarily, or.. whatever!! Anything!! Well, this is exactly, what my squid_dedup helper was created for! See my announcement: Subject: [squid-users] New StoreID helper: squid_dedup Date: Mon, 09 May 2016 23:56:45 +0200 My openSUSE environment is fetching _all_ updates with byte-ranges from many servers. Therefor, I created squid_dedup. Your specific config could look like this: /etc/squid/dedup/mozilla.conf: [mozilla] match: http\:\/\/download\.cdn\.mozilla\.net/(.*) replace: http://download.cdn.mozilla.net.%(intdomain)s/\1 fetch: true The fetch parameter is unique among the other StoreID helper (AFAIK): it is fetching the object after a certain delay with a pool of fetcher threads. The idea is: after the first access for an object, wait a bit (global setting, default: 15 secs), and then fetch the whole thing once. It won't solve anything for the first client, but for all subsequent accesses. The fetcher avoids fetching anything more than once by checking the http headers. This is a pretty new project, but be assured, that the basic functions are working fine, and I will do my best to solve any upcoming issues. It is implemented with Python3 and prepared for supporting additional features easily, while keeping a good part of an eye on efficiency. Let me know, if you're going to try it. Pete ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Getting the full file content on a range request, but not on EVERY get ...
On Wed, 2016-05-11 at 21:37 -0300, Heiler Bemerguy wrote: > > Hey guys, > First take a look at the log: > root@proxy:/var/log/squid# tail -f access.log |grep http://download.c > dn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar > 1463011781.572 8776 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 > application/octet-stream > 1463011851.008 9347 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463011920.683 9645 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.9 > application/octet-stream > 1463012000.144 19154 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463012072.276 12121 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463012145.643 13358 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463012217.472 11772 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463012294.676 17148 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > 1463012370.131 15272 10.1.3.236 TCP_MISS/206 300520 GET http://downl > oad.cdn.mozilla.net/pub/firefox/releases/45.0.1/update/win32/pt- > BR/firefox-45.0.1.complete.mar - HIER_DIRECT/200.216.8.32 > application/octet-stream > Now think: An user is just doing a segmented/ranged download, right? > Squid won't cache the file because it is a range-download, not a full > file download. > But I WANT squid to cache it. So I decide to use "range_offset_limit > -1", but then on every GET squid will re-download the file from the > beginning, opening LOTs of simultaneous connections and using too > much bandwidth, doing just the OPPOSITE it's meant to! > > Is there a smart way to allow squid to download it from the beginning > to the end (to actually cache it), but only on the FIRST request/get? > Even if it makes the user wait for the full download, or cancel it > temporarily, or.. whatever!! Anything!! > > Best Regards, > -- > Heiler Bemerguy - (91) 98151-4894 > Assessor Técnico - CINBESA (91) 3184-1751 > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users Hi, I believe, you describe the bug http://bugs.squid-cache.org/show_bu g.cgi?id=4469 I tried to reproduce the problem and have found that the problem appears only with rock storage configurations. Can you try with ufs/aufs storage? ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
