Sorry, the cacheurl log should be as -- 20150218.15h16m53s Adding pattern/replacement pair: 'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> 'http://video-srv.youtube.comi.atsinternal/$3.mp4' 20150218.15h17m02s Mismatch pattern:'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> URL:'www.youtube.com:443/'
20150218.15h17m02s Mismatch pattern:'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> URL:'manifest.googlevideo.com:443/' 20150218.15h17m09s Mismatch pattern:'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> URL:'www.youtube.com:443/' 20150218.15h17m09s Mismatch pattern:'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> URL:'manifest.googlevideo.com:443/' 20150218.15h17m09s Mismatch pattern:'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> URL:'r10---sn-nwj7knek.googlevideo.com:443/' --- From: Yue, Cong [mailto:[email protected]] Sent: Wednesday, February 18, 2015 3:07 PM To: [email protected] Subject: RE: cacheurl plugin does not work for youtube I add some debug code into cacheurl plug as follows, it shows the video URL for google is with very short version. There is no parameters in the URL any more. static int regex_substitute(char **buf, char *str, regex_info *info) { ... if (matchcount < 0) { switch (matchcount) { case PCRE_ERROR_NOMATCH: //TODO ADD MISMATCH URL OUTPUT HERE. if (log) { TSTextLogObjectWrite(log, "Mismatch pattern:'%s' -> URL:'%s'\n<URL:'%25s'\n>", info->pattern, str); } TSDebug(PLUGIN_NAME, "Mismatch pattern:'%s' -> URL:'%s'\n<URL:'%25s'\n>", info->pattern, str); break; default: The log is as 20150218.15h00m40s Mismatch pattern:'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)' -> URL:'r10---sn-nwj7knek.googlevideo.com:443/' 20150218.15h00m51s Mismatch pattern:'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)' -> URL:'clients4.google.com:443/' Is there any way to cache youtube video with TS? Please kindly advise! Thanks, Cong From: Yue, Cong [mailto:[email protected]] Sent: Wednesday, February 18, 2015 10:42 AM To: [email protected]<mailto:[email protected]> Subject: cacheurl plugin does not work for youtube Hi I am trying to make youtube be cached with forward proxy, but youtube URL can not be redirected. I configured /usr/local/libexec/trafficserver/cacheurl.config as --- http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/f4v/.*id=tudou.itemid\=([0-9]*).*<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d%5b%5e&%5d*/f4v/.*id=tudou.itemid/=(%5b0-9%5d*).*> http://www.tudou.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/flv/.*id=tudou.itemid\=([0-9]*).*<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d%5b%5e&%5d*/flv/.*id=tudou.itemid/=(%5b0-9%5d*).*> http://www.tudou.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/youku/.*/(.*-.*-.*-.*-[^?]*)(.*)<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/youku/.*/(.*-.*-.*-.*-%5b%5e?%5d*)(.*)> http://www.youku.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/sohu/[0-9]*/[0-9]*/[0-9]*/(.*).mp4?key=.*<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/sohu/%5b0-9%5d*/%5b0-9%5d*/%5b0-9%5d*/(.*).mp4?key=.*> http://tv.sohu.com/$1.mp4 http://.*\..*\..*\..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3}/([^?]*)(.*)<http://.*/..*/..*/..*/.*/.com/flvdownload/%5b%5b:digit:%5d%5d%7b1,3%7d/%5b%5b:digit:%5d%5d%7b1,3%7d/(%5b%5e?%5d*)(.*)> http://www.56.com/$1 http://[[:digit:]]{1,3}/mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,6}/(.*\.mp4)<http://[[:digit:]]%7b1,3%7d/mp4files/.*/.*/.com/images/tuiguang/%5b%5b:digit:%5d%5d%7b6,6%7d/(.*/.mp4)> http://www.56.com/$1 http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)<http://.*/..*/.com/images/tuiguang/(%5b%5b:digit:%5d%5d%7b6,6%7d)/(.*/.mp4)> http://www.56.com/tuiguang/$1/$2 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/mov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/mov.bn.netease.com/.*/.*/.*/.*/.*/(%5b%5e?%5d*)(.*)> http://v.163.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/.*-.*-.*/.*/cemov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)<http://[[:digit:]]%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.%5b%5b:digit:%5d%5d%7b1,3%7d/.*-.*-.*/.*/cemov.bn.netease.com/.*/.*/.*/.*/.*/(%5b%5e?%5d*)(.*)> http://v.163.com/$1 #YOUTUBE https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).* http://video-srv.youtube.comi.atsinternal/$3.mp4 --- >From /usr/local/var/log/trafficserver/cacheurl.log it shows for youku.com, the URL match can happen, but for youtube.com, it does not happen. I checked my url with online regular expression tool, it shows the URL of " " works. This is the log of /usr/local/var/log/trafficserver/cacheurl.log ---- 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/f4v/.*id=tudou.itemid\=([0-9]*).*' -> 'http://www.tudou.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/flv/.*id=tudou.itemid\=([0-9]*).*' -> 'http://www.tudou.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/youku/.*/(.*-.*-.*-.*-[^?]*)(.*)' -> 'http://www.youku.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/sohu/[0-9]*/[0-9]*/[0-9]*/(.*).mp4?key=.*' -> 'http://tv.sohu.com/$1.mp4' 20150218.10h24m54s Adding pattern/replacement pair: 'http://.*\..*\..*\..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3}/([^?]*)(.*)' -> 'http://www.56.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}/mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,6}/(.*\.mp4)' -> 'http://www.56.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)' -> 'http://www.56.com/tuiguang/$1/$2' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/mov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)' -> 'http://v.163.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/.*-.*-.*/.*/cemov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)' -> 'http://v. 163.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=[a-zA-Z0-9.\-\_]*).*' -> 'http://video-srv.youtube.comi.atsinternal/ $3.mp4' 20150218.10h26m22s Rewriting cache URL for http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=58632024139_23738066383&ns=164654 35_23569878&special=true to http://www.youku.com/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4 --------- The squid log is as follows. [root@ats1 trafficserver]# traffic_logcat squid.blog ----- 1424283922.032 880 10.0.0.45 TCP_MISS/200 105743 CONNECT r6---sn-vgqsenel.googlevideo.com:443/ - DIRECT/r6---sn-vgqsenel.googlevideo.com - 1424283927.747 17076 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283927.747 17076 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283927.748 17046 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht.com - 1424283927.748 17079 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283927.749 17079 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283927.749 17080 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283927.826 17155 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht.com - 1424283930.226 175 10.0.0.210 TCP_MISS/200 43699 CONNECT www.youtube.com:443/<http://www.youtube.com:443/> - DIRECT/www.youtube.com - 1424283930.386 99 10.0.0.210 TCP_MISS/200 7901 CONNECT manifest.googlevideo.com:443/ - DIRECT/manifest.googlevideo.com - 1424283941.575 30178 10.0.0.45 TCP_MISS/200 897 CONNECT gg.google.com:443/ - DIRECT/gg.google.com - 1424283941.576 30899 10.0.0.45 TCP_MISS/200 1107 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht.com - 1424283941.577 30906 10.0.0.45 TCP_MISS/200 2410 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com - 1424283942.571 30347 10.0.0.45 TCP_MISS/200 947 CONNECT ssl.gstatic.com:443/ - DIRECT/ssl.gstatic.com - 1424283970.120 39690 10.0.0.210 TCP_MISS/200 437526696 CONNECT r10---sn-a5m7lnel.googlevideo.com:443/ - DIRECT/r10---sn-a5m7lnel.googlevideo.com - 1424283980.655 671 10.0.0.45 TCP_MISS/200 252 GET 1424283983.193 12 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=76165731532_23738066400&ns=16576127_23459186&special=true - NONE/- video/mp4 1424283984.199 0 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=314386706919_23738066416&ns=16686819_23348494&special=true - NONE/- video/mp4 1424283985.253 4 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=410790030197_23738066434&ns=16797511_23237802&special=true - NONE/- video/mp4 1424283987.279 7 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=314386706950_23738066468&ns=17018895_23016418&special=true - NONE/- video/mp4 1424283997.579 9 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=410790030290_23738066639&ns=18125815_21909498&special=true - NONE/- video/mp4 ------- >From squid log, it seems I can not get full URL of youtube, but with >youtube-dl, I check the url should work with my regular expression. ----- [root@test-client1 webpages]# youtube-dl -v --proxy http://10.0.0.204:80 https://www.youtube.com/watch?v=q1mndAYZlio [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['-v', '--proxy', 'http://10.0.0.204:80', 'https://www.youtube.com/watch?v=q1mndAYZlio'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2015.02.18.1 [debug] Python version 2.6.6 - Linux-2.6.32-504.3.3.el6.x86_64-x86_64-with-centos-6.6-Final [debug] exe versions: none [debug] Proxy map: {u'http': 'http://10.0.0.204:80', u'https': 'http://10.0.0.204:80'} [youtube] q1mndAYZlio: Downloading webpage [youtube] q1mndAYZlio: Extracting video information [youtube] q1mndAYZlio: Downloading DASH manifest [debug] Invoking downloader on u'https://r10---sn-a5m7lnel.googlevideo.com/videoplayback?signature=78D8EB7D568039D930E67820E5D8751C67AD3273.68A236DF130AC81356A4970AFEFD7168E277111A&upn=ypVWJMpcfaA&mime=video%2Fmp4&initcwndbps=5183750&source=youtube&pl=18&sver=3&expire=1424305562&mm=31&dur=2857.052&id=o-AITnvpDOi0FfUY-UijIKeWK61KsdMQZSUd0E_NnKO8Od&itag=22&key=yt5&ip=208.184.212.172&fexp=902039%2C905657%2C927622%2C936109%2C9405708%2C9406015%2C9407010%2C943917%2C947225%2C948124%2C948807%2C952302%2C952605%2C952612%2C952901%2C955100%2C955301%2C957201%2C959701%2C960610&mt=1424283746&mv=m&ms=au&ratebypass=yes&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&ipbits=0&requiressl=yes' [download] Destination: Building a large scale CDN with Apache Traffic Server - Jan van Doorn-q1mndAYZlio.mp4 [download] 100% of 416.52MiB in 00:39 ------ Can somebody kindly advise? Thanks, Cong ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.
