[
https://issues.apache.org/jira/browse/TS-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolai Gorchilov updated TS-3995:
----------------------------------
Description:
Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with
proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live
streaming.
The actual problem is that ATS rewrites origin response from 200 to 304, due to
If-Modified-Since conditional header in client's request. As per ATS logic
object is unmodified, but in fact it is. Most probably it's related to the fact
that player and server somehow play with if-modified-since/last-modified
headers pair to communicate position in the live stream.
As result, Akamai player keeps repeating the said request, expecting it's 200,
but getting 304 thus live video freezes forever, just a few seconds after start.
IMHO when proxy.config.http.cache.when_to_revalidate = 4, ATS shall not
interfere with origin response in this manner.
Here's a debug log of request and response headers at different states in a
single transaction:
{noformat}
+++++++++ Incoming Request +++++++++
-- State Machine Id: 168
GET
http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Proxy's Request +++++++++
-- State Machine Id: 168
GET
/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Proxy's Request after hooks +++++++++
-- State Machine Id: 168
GET
/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Incoming O.S. Response +++++++++
-- State Machine Id: 168
HTTP/1.1 200 OK
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: video/abst
Content-Length: 122
Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 05 Nov 2015 11:38:01 GMT
Connection: keep-alive
Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
domain=sshds5-lh.akamaihd.net
+++++++++ Base Header for Building Response +++++++++
-- State Machine Id: 168
HTTP/1.1 200 OK
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: video/abst
Content-Length: 122
Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 05 Nov 2015 11:38:01 GMT
Connection: keep-alive
Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
domain=sshds5-lh.akamaihd.net
+++++++++ Proxy's Response 2 +++++++++
-- State Machine Id: 168
HTTP/1.1 304 Not Modified
Date: Thu, 05 Nov 2015 11:38:01 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Connection: keep-alive
+++++++++ Proxy's Response (Client Conditionals) +++++++++
-- State Machine Id: 168
HTTP/1.1 304 Not Modified
Date: Thu, 05 Nov 2015 11:38:01 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Connection: keep-alive
{noformat}
was:
Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with
proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live
streaming by overwriting non-cacheable origin response from 200 to 304 received
against conditional request originating from the client.
Here's a log of all request and response headers for a single transaction,
collected via debug:
{noformat}
+++++++++ Incoming Request +++++++++
-- State Machine Id: 168
GET
http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Proxy's Request +++++++++
-- State Machine Id: 168
GET
/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Proxy's Request after hooks +++++++++
-- State Machine Id: 168
GET
/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
HTTP/1.1
Host: sshds5-lh.akamaihd.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
X-Requested-With: ShockwaveFlash/19.0.0.226
Accept: */*
DNT: 1
Referer: http://www.hotstar.com/
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.8,bg;q=0.6
Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
+++++++++ Incoming O.S. Response +++++++++
-- State Machine Id: 168
HTTP/1.1 200 OK
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: video/abst
Content-Length: 122
Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 05 Nov 2015 11:38:01 GMT
Connection: keep-alive
Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
domain=sshds5-lh.akamaihd.net
+++++++++ Base Header for Building Response +++++++++
-- State Machine Id: 168
HTTP/1.1 200 OK
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: video/abst
Content-Length: 122
Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 05 Nov 2015 11:38:01 GMT
Connection: keep-alive
Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
domain=sshds5-lh.akamaihd.net
+++++++++ Proxy's Response 2 +++++++++
-- State Machine Id: 168
HTTP/1.1 304 Not Modified
Date: Thu, 05 Nov 2015 11:38:01 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Connection: keep-alive
+++++++++ Proxy's Response (Client Conditionals) +++++++++
-- State Machine Id: 168
HTTP/1.1 304 Not Modified
Date: Thu, 05 Nov 2015 11:38:01 GMT
Expires: Thu, 05 Nov 2015 11:38:01 GMT
Cache-Control: max-age=0, no-cache
Connection: keep-alive
{noformat}
> "[hcoofsr] conditional request, 200 response, send back 304 if possible
> [crc=304]" breaks akamaihd.net live streaming
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: TS-3995
> URL: https://issues.apache.org/jira/browse/TS-3995
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache, Core, HTTP
> Affects Versions: 5.3.2
> Reporter: Nikolai Gorchilov
>
> Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with
> proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live
> streaming.
> The actual problem is that ATS rewrites origin response from 200 to 304, due
> to If-Modified-Since conditional header in client's request. As per ATS logic
> object is unmodified, but in fact it is. Most probably it's related to the
> fact that player and server somehow play with if-modified-since/last-modified
> headers pair to communicate position in the live stream.
> As result, Akamai player keeps repeating the said request, expecting it's
> 200, but getting 304 thus live video freezes forever, just a few seconds
> after start.
> IMHO when proxy.config.http.cache.when_to_revalidate = 4, ATS shall not
> interfere with origin response in this manner.
> Here's a debug log of request and response headers at different states in a
> single transaction:
> {noformat}
> +++++++++ Incoming Request +++++++++
> -- State Machine Id: 168
> GET
> http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
> HTTP/1.1
> Host: sshds5-lh.akamaihd.net
> Connection: keep-alive
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
> X-Requested-With: ShockwaveFlash/19.0.0.226
> Accept: */*
> DNT: 1
> Referer: http://www.hotstar.com/
> Accept-Encoding: gzip
> Accept-Language: en-US,en;q=0.8,bg;q=0.6
> Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
> hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
> If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
> +++++++++ Proxy's Request +++++++++
> -- State Machine Id: 168
> GET
> /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
> HTTP/1.1
> Host: sshds5-lh.akamaihd.net
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
> X-Requested-With: ShockwaveFlash/19.0.0.226
> Accept: */*
> DNT: 1
> Referer: http://www.hotstar.com/
> Accept-Encoding: gzip
> Accept-Language: en-US,en;q=0.8,bg;q=0.6
> Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
> hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
> If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
> +++++++++ Proxy's Request after hooks +++++++++
> -- State Machine Id: 168
> GET
> /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44
> HTTP/1.1
> Host: sshds5-lh.akamaihd.net
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
> X-Requested-With: ShockwaveFlash/19.0.0.226
> Accept: */*
> DNT: 1
> Referer: http://www.hotstar.com/
> Accept-Encoding: gzip
> Accept-Language: en-US,en;q=0.8,bg;q=0.6
> Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
> hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8
> If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT
> +++++++++ Incoming O.S. Response +++++++++
> -- State Machine Id: 168
> HTTP/1.1 200 OK
> Server: AkamaiGHost
> Mime-Version: 1.0
> Content-Type: video/abst
> Content-Length: 122
> Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
> Expires: Thu, 05 Nov 2015 11:38:01 GMT
> Cache-Control: max-age=0, no-cache
> Pragma: no-cache
> Date: Thu, 05 Nov 2015 11:38:01 GMT
> Connection: keep-alive
> Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
> path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
> domain=sshds5-lh.akamaihd.net
> +++++++++ Base Header for Building Response +++++++++
> -- State Machine Id: 168
> HTTP/1.1 200 OK
> Server: AkamaiGHost
> Mime-Version: 1.0
> Content-Type: video/abst
> Content-Length: 122
> Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT
> Expires: Thu, 05 Nov 2015 11:38:01 GMT
> Cache-Control: max-age=0, no-cache
> Pragma: no-cache
> Date: Thu, 05 Nov 2015 11:38:01 GMT
> Connection: keep-alive
> Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==;
> path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/;
> domain=sshds5-lh.akamaihd.net
> +++++++++ Proxy's Response 2 +++++++++
> -- State Machine Id: 168
> HTTP/1.1 304 Not Modified
> Date: Thu, 05 Nov 2015 11:38:01 GMT
> Expires: Thu, 05 Nov 2015 11:38:01 GMT
> Cache-Control: max-age=0, no-cache
> Connection: keep-alive
> +++++++++ Proxy's Response (Client Conditionals) +++++++++
> -- State Machine Id: 168
> HTTP/1.1 304 Not Modified
> Date: Thu, 05 Nov 2015 11:38:01 GMT
> Expires: Thu, 05 Nov 2015 11:38:01 GMT
> Cache-Control: max-age=0, no-cache
> Connection: keep-alive
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
