I've been running daily load tests for almost the last 6+ months. 6000 simulated users, which is 12k connections at around 17-18gbps. Before I started playing with the collapse_forwarding plugin, I would see requests 'leaking' to the origin, when those requests should have been served by cache. Now that I have collapse_forwarding tuned to my testing environment, combined with the associated records.config parameters, I no longer see 'leaking' origin requests. This goes for both 100% cached HLS VOD(v-segs=2.7MB a-segs=66KB) and for simulated HLS live.
On Wed, Mar 7, 2018 at 8:58 AM, BURAK EGERCI <[email protected]> wrote: > Hi, > > Thanks for the quick response. > > > > I m using m3u8 file. But not the main m3u8 as I want to specifiy the HLS > profile. Lets assume index.m3u8 includes 01.m3u8 and 02.m3u8, I m > requesting 01.m3u8. > > > > Once ~4000 concurrent clients are accessing, the issue can be visible. It > is not visible if the accessing client number is less than 4000. > > > > Currently Ram storage is configured. > > > > Thanks for Collapse Forwarding algorithm, I will set this up, that may > solve the issue. Will let you know after trial. > > Here is the records.config; > > > > CONFIG proxy.config.accept_threads INT 5 > > CONFIG proxy.config.admin.admin_user STRING admin > > CONFIG proxy.config.admin.autoconf_port INT 8083 > > CONFIG proxy.config.admin.number_config_bak INT 3 > > CONFIG proxy.config.admin.user_id STRING ats > > CONFIG proxy.config.alarm.abs_path STRING NULL > > CONFIG proxy.config.alarm.bin STRING example_alarm_bin.sh > > CONFIG proxy.config.alarm_email STRING ats > > CONFIG proxy.config.allocator.debug_filter INT 0 > > CONFIG proxy.config.allocator.dontdump_iobuffers INT 0 > > CONFIG proxy.config.allocator.enable_reclaim INT 0 > > CONFIG proxy.config.allocator.hugepages INT 0 > > CONFIG proxy.config.allocator.max_overage INT 3 > > CONFIG proxy.config.allocator.thread_freelist_size INT 1024 > > CONFIG proxy.config.body_factory.enable_customizations INT 1 > > CONFIG proxy.config.body_factory.enable_logging INT 0 > > CONFIG proxy.config.body_factory.response_suppression_mode INT 0 > > CONFIG proxy.config.body_factory.template_sets_dir STRING > etc/trafficserver/body_factory > > CONFIG proxy.config.cache.control.filename STRING cache.config > > CONFIG proxy.config.cache.enable_read_while_writer INT 1 > > CONFIG proxy.config.cache.hostdb.sync_frequency INT 0 > > CONFIG proxy.config.cache.hosting_filename STRING hosting.config > > CONFIG proxy.config.cache.http.compatibility.4-2-0-fixup INT 0 > > CONFIG proxy.config.cache.limits.http.max_alts INT 5 > > CONFIG proxy.config.cache.max_doc_size INT 0 > > CONFIG proxy.config.cache.min_average_object_size INT 131072 > > CONFIG proxy.config.cache.mutex_retry_delay INT 2 > > CONFIG proxy.config.cache.permit.pinning INT 0 > > CONFIG proxy.config.cache.ram_cache.algorithm INT 1 > > CONFIG proxy.config.cache.ram_cache.compress INT 0 > > CONFIG proxy.config.cache.ram_cache.size INT 34359738368 > > CONFIG proxy.config.cache.ram_cache.use_seen_filter INT 1 > > CONFIG proxy.config.cache.ram_cache_cutoff INT 1073741824 > > CONFIG proxy.config.cache.target_fragment_size INT 1048576 > > CONFIG proxy.config.cache.threads_per_disk INT 8 > > CONFIG proxy.config.cluster.cluster_configuration STRING cluster.config > > CONFIG proxy.config.cluster.cluster_port INT 8086 > > CONFIG proxy.config.cluster.ethernet_interface STRING lo > > CONFIG proxy.config.cluster.log_bogus_mc_msgs INT 1 > > CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37 > > CONFIG proxy.config.cluster.mc_ttl INT 1 > > CONFIG proxy.config.cluster.mcport INT 8089 > > CONFIG proxy.config.cluster.rsport INT 8088 > > CONFIG proxy.config.config_dir STRING etc/trafficserver > > CONFIG proxy.config.core_limit INT -1 > > CONFIG proxy.config.diags.debug.enabled INT 0 > > CONFIG proxy.config.diags.debug.tags STRING http.*|dns.* > > CONFIG proxy.config.diags.show_location INT 0 > > CONFIG proxy.config.dns.max_dns_in_flight INT 2048 > > CONFIG proxy.config.dns.nameservers STRING NULL > > CONFIG proxy.config.dns.resolv_conf STRING /etc/resolv.conf > > CONFIG proxy.config.dns.round_robin_nameservers INT 0 > > CONFIG proxy.config.dns.search_default_domains INT 0 > > CONFIG proxy.config.dns.splitDNS.enabled INT 0 > > CONFIG proxy.config.dns.url_expansions STRING NULL > > CONFIG proxy.config.dns.validate_query_name INT 0 > > CONFIG proxy.config.dump_mem_info_frequency INT 0 > > CONFIG proxy.config.env_prep STRING example_prep.sh > > Thanks & kind regards, > > Burak > > > > > > *From:* Vasanth Mathivanan [mailto:[email protected]] > *Sent:* Wednesday, March 7, 2018 5:09 PM > *To:* [email protected] > *Subject:* COMMERCIAL:RE: HLS streaming issue > > > > Hi, > > I have Lot of Questions In front of You: > > > > 1.Are You using M3u8 or Direct ts file ? > > 2.How Many Clients Accessing ( because One client to hits Directly from > origin the Other Clients will be loaded on Cache ? > > 3.Which Storage Method Using like Ram or Hard Disk ? > > 4.Which Algorithm Using in hdr_rw_config file? > > 5.Some times happen multiple Concurrent requests so you do need to add > Collapse Forwarding Algorithm > > 6.What Parameter using in records. Config file ? > > > > Thanks & regards > > > > Vasanth M > > > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > *From: *BURAK EGERCI <[email protected]> > *Sent: *Wednesday, March 7, 2018 7:09 PM > *To: *[email protected] > *Subject: *HLS streaming issue > > > > > > Dear All, > > > > We are working on HLS streaming. Once # of concurrent sessions reach > around 4K, we are experiencing issues regarding the cache algorithm. > > > > Although Cache.config is set to 10minutes for ts files ( > url_regex=(.*)\.(ts)$ ttl-in-cache=10m ), lots of requests will be directed > to origin server which previously get http 200 from origin server, ATS > continues to direct the request to origin server. > > > > Have anyone experienced such an issue? Or does anyone have an idea how to > solve this issue? > > > > *Please see origin server log below…* ATS sending lots of requests for > the same file at the same time although it gets http 200. > > > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 181000 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 78192 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > 176.235.11.9 - - [07/Mar/2018:15:18:04 +0300] "GET > /Fashionbox/20180125T023356-06-357677.ts HTTP/1.1" 200 665896 "-" > "User-Agent" > > > > *Please see ATS log at the same time; * > > > > 1520425086.244 chi=52.214.59.217 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=39879 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.254 chi=34.245.110.191 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40006 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.257 chi=34.242.200.72 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40043 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.258 chi=34.253.240.24 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=39944 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.259 chi=54.171.6.29 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=39951 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.259 chi=34.245.98.243 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40047 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.411 chi=34.245.117.91 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40020 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.414 chi=34.253.239.40 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40150 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.421 chi=34.243.255.27 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40176 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.427 chi=34.249.94.171 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40083 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.428 chi=52.48.159.168 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40028 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.431 chi=54.246.245.203 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=38984 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.435 chi=34.249.44.167 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40103 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.435 chi=54.171.125.63 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=37816 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > 1520425086.435 chi=54.171.125.63 phn=tvgbzcdnes02 php=80 shn=176.235.11.2 > url=http://tvgbztcdnes02.channels.turkcelltvplus.com. > tr/Fashionbox/20180125T023356-06-357677.ts cqhm=GET cqhv=HTTP/1.1 > pssc=200 ttms=40198 b=665896 sssc=200 sscl=665896 cfsc=FIN pfsc=FIN > crc=TCP_MISS phr=DIRECT pqsn=176.235.11.2 uas="User-Agent" > > > > > > Thanks & kind regards, > > Burak > > > > <http://turkcell.li/iyaani> > > Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi > tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi > icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin > icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz > kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir. > > TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu > konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne > sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve > saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen > kisiye aittir ve TURKCELLin goruslerini yansitmayabilir > > Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir. > ------------------------------ > > This e-mail and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom they are > addressed. If you are not the intended recipient you are hereby notified > that any dissemination, forwarding, copying or use of any of the > information is strictly prohibited, and the e-mail should immediately be > deleted. > > TURKCELL makes no warranty as to the accuracy or completeness of any > information contained in this message and hereby excludes any liability of > any kind for the information contained therein or for the information > transmission, reception, storage or use of such in any way whatsoever. The > opinions expressed in this message belong to sender alone and may not > necessarily reflect the opinions of TURKCELL. > > This e-mail has been scanned for all known computer viruses. >
