Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, Oct 14, 2009 at 1:04 PM, Andrew Scherkus scher...@google.comwrote: We use a combination of in-memory and block-based caching for media resources. There is no guarantee whatsoever on what is loaded. There's a nice side benefit of allowing complete random access to the file if the server supports range request (in my opinion a must if you're working on an audio/video site, but that's a side issue). Indeed, we support that too. For http:// we don't support load at all and never reach NETWORK_LOADED. We're also baffled at what to report in our progress events because the numbers are seemingly useless in our context. Sometimes we'll cache the end of the file to grab some useful metadata, but we don't want to fire a progress event saying we've loaded the last bit of the file... We have the same issue, unsurprisingly. Currently our progress events are not very useful because we report the download position (although we do suppress progress events while we read metadata from the end of the file). According to the spec, however, progress events should report the *amount* of data downloaded, not the position within the resource. That's a bit more useful, although it's still unclear what to do when data is discarded from the cache and then re-downloaded. Being the only browser (I think?) that doesn't fire the load event we've ran into site compatibility issues. I believe a lot of web authors are used to tacking onload= into a tag and expecting it to work. I've reached out to web authors to educate them about using canplaytype() + oncanplay/oncanplaythrough as alternatives to user agent checking and onload. Good, thanks. We plan to stop firing the load event in the next (and all future) versions of Gecko, so we'll be doing that outreach too. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 8 Oct 2009, Robert O'Callahan wrote: http://www.whatwg.org/specs/web-apps/current-work/#loading-the-media-resource In the resource fetch algorithm, after we reach the NETWORK_LOADED state in step 3 which indicates that all the data we need to play the resource is now available locally, we end the resource fetch algorithm. However, in Gecko we have a media cache which might discard blocks of media data after we've reached the NETWORK_LOADED state (to make room for data for other loading resources). This means we might have to start fetching the resource again later. The spec does not seem to allow for this. Do we need to change our behavior, or does the spec need to change to accommodate our behavior? I'd prefer not to change our behavior since I think to follow the spec we'd need to pin the entire resource permanently in the cache after we reached NETWORK_LOADED, which could be highly suboptimal in some situations. Step 2 says While the user agent might still need network access to obtain parts of the media resource, the user agent must remain on this step. It even has an example of a UA that discards video data. Another issue is that it's not completely clear to me what is meant by While the user agent might still need network access to obtain parts of the media resource What if there is data in the resource that we don't need in order to play through normally, but which might be needed in some special situations (e.g., enabling subtitles, or seeking using an index), and we optimize to not load that data unless/until we need it? In that case would we never reach NETWORK_LOADED? Correct. In general NETWORK_LOADED and the load event seem rather useless and dangerous IMHO. If you're playing a resource that doesn't fit in your cache then you'll certainly never reach NETWORK_LOADED, and since authors can't know the cache size they can never rely on load firing. And if you allow the cache discarding behavior I described above, authors can't rely on data actually being present locally even after load has fired. I suspect many authors will make invalid assumptions about load being sure to fire and about what load means if it does fire. Does anyone have any use cases that load actually solves? I'm happy to remove 'load'. It's mostly a holdover from the first cut at this API, where this wasn't thought through in much detail. On Fri, 9 Oct 2009, Robert O'Callahan wrote: Me too. I'm game if you are! So I propose: 1) Remove the NETWORK_LOADED state and load and loadend events from media elements. The resource fetch algorithm simply never transitions from step 2 to step 3. Done. NOTE: I've changed NETWORK_NO_SOURCE's value from 4 to 3! On Fri, 9 Oct 2009, Simon Pieters wrote: 'loadend' also fires after 'abort' and (when using the src attribute) 'error'. Should that stay as is, or also be removed? On Fri, 9 Oct 2009, Philip Jägenstedt wrote: Since we're going to contradict the progress events spec anyway, I would suggest dropping all 'loadend' events. They're just not very useful. I've left it in the other cases, since, well, Progress Events says to. But I'd be happy to drop loadend in all cases (including other Progress Events cases) if that makes sense. On Tue, 13 Oct 2009, Andrew Scherkus wrote: We're also baffled at what to report in our progress events because the numbers are seemingly useless in our context. Sometimes we'll cache the end of the file to grab some useful metadata, but we don't want to fire a progress event saying we've loaded the last bit of the file... On Wed, 14 Oct 2009, Robert O'Callahan wrote: We have the same issue, unsurprisingly. Currently our progress events are not very useful because we report the download position (although we do suppress progress events while we read metadata from the end of the file). According to the spec, however, progress events should report the *amount* of data downloaded, not the position within the resource. That's a bit more useful, although it's still unclear what to do when data is discarded from the cache and then re-downloaded. Ok, I've switched them back to regular Event events, though using the same event names. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, 14 Oct 2009 12:08:19 +0200, Ian Hickson i...@hixie.ch wrote: On Fri, 9 Oct 2009, Philip Jägenstedt wrote: Since we're going to contradict the progress events spec anyway, I would suggest dropping all 'loadend' events. They're just not very useful. I've left it in the other cases, since, well, Progress Events says to. But I'd be happy to drop loadend in all cases (including other Progress Events cases) if that makes sense. On Wed, 14 Oct 2009, Robert O'Callahan wrote: We have the same issue, unsurprisingly. Currently our progress events are not very useful because we report the download position (although we do suppress progress events while we read metadata from the end of the file). According to the spec, however, progress events should report the *amount* of data downloaded, not the position within the resource. That's a bit more useful, although it's still unclear what to do when data is discarded from the cache and then re-downloaded. Ok, I've switched them back to regular Event events, though using the same event names. We added loadend just to comply with Progress Events. Now that we fire simple events instead, please drop loadend again as it serves no purpose at all. I doubt any browser has yet shipped an implementation firing loadend, correct me if I'm wrong. -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, 14 Oct 2009 12:51:09 +0200, Philip Jägenstedt phil...@opera.com wrote: We added loadend just to comply with Progress Events. Now that we fire simple events instead, please drop loadend again as it serves no purpose at all. I doubt any browser has yet shipped an implementation firing loadend, correct me if I'm wrong. The loadend event is dispatched for XMLHttpRequest and that might be in Gecko I suppose. I'm not particularly attached to it though. I just added it for consistency with the media elements and the progress events specification. -- Anne van Kesteren http://annevankesteren.nl/
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, 14 Oct 2009 12:58:17 +0200, Anne van Kesteren ann...@opera.com wrote: On Wed, 14 Oct 2009 12:51:09 +0200, Philip Jägenstedt phil...@opera.com wrote: We added loadend just to comply with Progress Events. Now that we fire simple events instead, please drop loadend again as it serves no purpose at all. I doubt any browser has yet shipped an implementation firing loadend, correct me if I'm wrong. The loadend event is dispatched for XMLHttpRequest and that might be in Gecko I suppose. I'm not particularly attached to it though. I just added it for consistency with the media elements and the progress events specification. I'm only talking about dropping it for media elements. -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, 14 Oct 2009, Philip J�genstedt wrote: On Wed, 14 Oct 2009 12:08:19 +0200, Ian Hickson i...@hixie.ch wrote: On Fri, 9 Oct 2009, Philip J�genstedt wrote: Since we're going to contradict the progress events spec anyway, I would suggest dropping all 'loadend' events. They're just not very useful. I've left it in the other cases, since, well, Progress Events says to. But I'd be happy to drop loadend in all cases (including other Progress Events cases) if that makes sense. On Wed, 14 Oct 2009, Robert O'Callahan wrote: We have the same issue, unsurprisingly. Currently our progress events are not very useful because we report the download position (although we do suppress progress events while we read metadata from the end of the file). According to the spec, however, progress events should report the *amount* of data downloaded, not the position within the resource. That's a bit more useful, although it's still unclear what to do when data is discarded from the cache and then re-downloaded. Ok, I've switched them back to regular Event events, though using the same event names. We added loadend just to comply with Progress Events. Now that we fire simple events instead, please drop loadend again as it serves no purpose at all. I doubt any browser has yet shipped an implementation firing loadend, correct me if I'm wrong. Ok, removed loadend from HTML5 altogether. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Wed, Oct 14, 2009 at 11:51 PM, Philip Jägenstedt phil...@opera.comwrote: We added loadend just to comply with Progress Events. Now that we fire simple events instead, please drop loadend again as it serves no purpose at all. I doubt any browser has yet shipped an implementation firing loadend, correct me if I'm wrong. We have never fired loadend. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
Chiming in here from Chrome's perspective... We use a combination of in-memory and block-based caching for media resources. There is no guarantee whatsoever on what is loaded. There's a nice side benefit of allowing complete random access to the file if the server supports range request (in my opinion a must if you're working on an audio/video site, but that's a side issue). For http:// we don't support load at all and never reach NETWORK_LOADED. We're also baffled at what to report in our progress events because the numbers are seemingly useless in our context. Sometimes we'll cache the end of the file to grab some useful metadata, but we don't want to fire a progress event saying we've loaded the last bit of the file... For file:// we immediately transition to NETWORK_LOADED. This is how we pass the WebKit layout tests :) Being the only browser (I think?) that doesn't fire the load event we've ran into site compatibility issues. I believe a lot of web authors are used to tacking onload= into a tag and expecting it to work. I've reached out to web authors to educate them about using canplaytype() + oncanplay/oncanplaythrough as alternatives to user agent checking and onload. I'm in favour of getting rid of both load and progress events. Andrew On Fri, Oct 9, 2009 at 2:27 AM, Robert O'Callahan rob...@ocallahan.orgwrote: On Fri, Oct 9, 2009 at 8:32 PM, Philip Jägenstedt phil...@opera.comwrote: Aesthetics is not a serious argument. More importantly, the progress events spec [1] requires that exactly one of error/abort/load be fired followed by loadend. Dropping load and loadend would be a willful violations of that spec. In my opinion, the progress events spec should be the one to change. [1] http://www.w3.org/TR/progress-events/ I agree. That spec says If the Operation successfully completes, the user agent *must* dispatch a load event. Here we're just dealing with an operation that never completes. The only real problem in the spec is that it says Exactly one of these *must* be dispatched, which seems to just not have considered the possibility of operations that run indefinitely. Some other Mozilla developers have actually argued that progress events in general don't make sense for media elements. The 'buffered' TimeRanges attribute gives you much more accurate and useful information than progress events. The progress event 'loaded' and 'total' attributes don't make a lot of sense in implementations where data may be discarded and redownloaded during the load. (If you discard some data, does the next progress event have a smaller 'loaded' value than the last one? Or does 'total' increase by the size of the discarded data?) But I don't want to open that can of worms just yet :-). Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 08 Oct 2009 23:19:28 +0200, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Oct 9, 2009 at 6:42 AM, Eric Carlson eric.carl...@apple.com wrote: I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it (including many of the WebKit layout tests :-( ), but I think we will be better off in the long run if we get rid of the event and network state now. Me too. I'm game if you are! So I propose: 1) Remove the NETWORK_LOADED state and load and loadend events from media elements. 'loadend' also fires after 'abort' and (when using the src attribute) 'error'. Should that stay as is, or also be removed? The resource fetch algorithm simply never transitions from step 2 to step 3. 2) Modify Gecko and Webkit accordingly. If we do part 2, which I think is already permitted by the spec, then authors will stop depending on load whether or not we get consensus for altering the spec. Rob -- Simon Pieters Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 08 Oct 2009 23:08:32 +0200, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Oct 9, 2009 at 1:32 AM, Philip Jägenstedt phil...@opera.com wrote: The spec notes that Some resources, e.g. streaming Web radio, can never reach the NETWORK_LOADED state. In my understanding, you mustn't go to NETWORK_LOADED if you can't guarantee that the resource will remain in cache. Browsers with clever caching or small caches simply won't send a load event most of the time. Right, HTML5 allows Gecko to simply never enter the NETWORK_LOADED state and send a load event. That would be the simplest thing for us to do, and I think the best in terms of letting us do intelligent caching. But I'd prefer not to do it alone. No objection to dropping the event and implementing as such here. Aesthetically, however, I think it would be strange to not have the load event. If you mean because of the analogy with the load events of other elements, Aesthetics is not a serious argument. More importantly, the progress events spec [1] requires that exactly one of error/abort/load be fired followed by loadend. Dropping load and loadend would be a willful violations of that spec. In my opinion, the progress events spec should be the one to change. [1] http://www.w3.org/TR/progress-events/ -- Philip Jägenstedt Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, 09 Oct 2009 08:42:12 +0200, Simon Pieters sim...@opera.com wrote: On Thu, 08 Oct 2009 23:19:28 +0200, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Oct 9, 2009 at 6:42 AM, Eric Carlson eric.carl...@apple.com wrote: I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it (including many of the WebKit layout tests :-( ), but I think we will be better off in the long run if we get rid of the event and network state now. Me too. I'm game if you are! So I propose: 1) Remove the NETWORK_LOADED state and load and loadend events from media elements. 'loadend' also fires after 'abort' and (when using the src attribute) 'error'. Should that stay as is, or also be removed? Since we're going to contradict the progress events spec anyway, I would suggest dropping all 'loadend' events. They're just not very useful. The resource fetch algorithm simply never transitions from step 2 to step 3. 2) Modify Gecko and Webkit accordingly. If we do part 2, which I think is already permitted by the spec, then authors will stop depending on load whether or not we get consensus for altering the spec. Rob -- Philip Jägenstedt Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 08 Oct 2009 22:53:45 +0200, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Oct 9, 2009 at 7:31 AM, Philip Jägenstedt phil...@opera.com wrote: I wouldn't be particularly opposed to dropping the load event, unless there's a use case for the guarantee that the resource won't be evicted from cache as long as the page is loaded. If there is a need for that, then I think we should add some kind of pinning API, although I'm not sure how it would avoid the problem of greedy authors pinning resources unnecessarily. But we'd have to consider the actual use cases. I'd say that the autobuffer attribute is exactly that API, which gives both author control and browser liberty to ignore it when necessary. -- Philip Jägenstedt Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, Oct 9, 2009 at 8:32 PM, Philip Jägenstedt phil...@opera.com wrote: Aesthetics is not a serious argument. More importantly, the progress events spec [1] requires that exactly one of error/abort/load be fired followed by loadend. Dropping load and loadend would be a willful violations of that spec. In my opinion, the progress events spec should be the one to change. [1] http://www.w3.org/TR/progress-events/ I agree. That spec says If the Operation successfully completes, the user agent *must* dispatch a load event. Here we're just dealing with an operation that never completes. The only real problem in the spec is that it says Exactly one of these *must* be dispatched, which seems to just not have considered the possibility of operations that run indefinitely. Some other Mozilla developers have actually argued that progress events in general don't make sense for media elements. The 'buffered' TimeRanges attribute gives you much more accurate and useful information than progress events. The progress event 'loaded' and 'total' attributes don't make a lot of sense in implementations where data may be discarded and redownloaded during the load. (If you discard some data, does the next progress event have a smaller 'loaded' value than the last one? Or does 'total' increase by the size of the discarded data?) But I don't want to open that can of worms just yet :-). Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
[whatwg] Restarting the media element resource fetch algorithm after load event
http://www.whatwg.org/specs/web-apps/current-work/#loading-the-media-resource In the resource fetch algorithm, after we reach the NETWORK_LOADED state in step 3 which indicates that all the data we need to play the resource is now available locally, we end the resource fetch algorithm. However, in Gecko we have a media cache which might discard blocks of media data after we've reached the NETWORK_LOADED state (to make room for data for other loading resources). This means we might have to start fetching the resource again later. The spec does not seem to allow for this. Do we need to change our behavior, or does the spec need to change to accommodate our behavior? I'd prefer not to change our behavior since I think to follow the spec we'd need to pin the entire resource permanently in the cache after we reached NETWORK_LOADED, which could be highly suboptimal in some situations. Another issue is that it's not completely clear to me what is meant by While the user agent might still need network access to obtain parts of the media resourcehttp://www.whatwg.org/specs/web-apps/current-work/#media-resource... What if there is data in the resource that we don't need in order to play through normally, but which might be needed in some special situations (e.g., enabling subtitles, or seeking using an index), and we optimize to not load that data unless/until we need it? In that case would we never reach NETWORK_LOADED? In general NETWORK_LOADED and the load event seem rather useless and dangerous IMHO. If you're playing a resource that doesn't fit in your cache then you'll certainly never reach NETWORK_LOADED, and since authors can't know the cache size they can never rely on load firing. And if you allow the cache discarding behavior I described above, authors can't rely on data actually being present locally even after load has fired. I suspect many authors will make invalid assumptions about load being sure to fire and about what load means if it does fire. Does anyone have any use cases that load actually solves? Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 08 Oct 2009 12:10:01 +0200, Robert O'Callahan rob...@ocallahan.org wrote: http://www.whatwg.org/specs/web-apps/current-work/#loading-the-media-resource In the resource fetch algorithm, after we reach the NETWORK_LOADED state in step 3 which indicates that all the data we need to play the resource is now available locally, we end the resource fetch algorithm. However, in Gecko we have a media cache which might discard blocks of media data after we've reached the NETWORK_LOADED state (to make room for data for other loading resources). This means we might have to start fetching the resource again later. The spec does not seem to allow for this. Do we need to change our behavior, or does the spec need to change to accommodate our behavior? I'd prefer not to change our behavior since I think to follow the spec we'd need to pin the entire resource permanently in the cache after we reached NETWORK_LOADED, which could be highly suboptimal in some situations. The spec notes that Some resources, e.g. streaming Web radio, can never reach the NETWORK_LOADED state. In my understanding, you mustn't go to NETWORK_LOADED if you can't guarantee that the resource will remain in cache. Browsers with clever caching or small caches simply won't send a load event most of the time. Another issue is that it's not completely clear to me what is meant by While the user agent might still need network access to obtain parts of the media resourcehttp://www.whatwg.org/specs/web-apps/current-work/#media-resource... What if there is data in the resource that we don't need in order to play through normally, but which might be needed in some special situations (e.g., enabling subtitles, or seeking using an index), and we optimize to not load that data unless/until we need it? In that case would we never reach NETWORK_LOADED? As I understand it, NETWORK_LOADED means that all bytes of the resource have been loaded, regardless of whether they will be used or not. Are there any formats that would actually allow not downloading parts of the resource in a meaningful way? Subtitles and indexes are too small to bother, and multiplexed audio/video tracks can hardly be skipped without zillions of HTTP Range requests. It seems to me that kind of thing would have to be done either with a server side media fragment request (using the 'track' dimension) or with an external audio/video track somehow synced to the master track (much like external subtitles). In general NETWORK_LOADED and the load event seem rather useless and dangerous IMHO. If you're playing a resource that doesn't fit in your cache then you'll certainly never reach NETWORK_LOADED, and since authors can't know the cache size they can never rely on load firing. And if you allow the cache discarding behavior I described above, authors can't rely on data actually being present locally even after load has fired. I suspect many authors will make invalid assumptions about load being sure to fire and about what load means if it does fire. Does anyone have any use cases that load actually solves? I agree, sites that depend on the load event sites will likely break randomly for file sizes that usually barely fit into the cache of the browser they were tested with. If browsers are conservative with bandwidth and only send the load event when it's true, I think we will have less of a problem however. Note that the load event isn't strictly needed, waiting for a progress event with loaded==total would achieve the same thing. Aesthetically, however, I think it would be strange to not have the load event. -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Oct 8, 2009, at 5:32 AM, Philip Jägenstedt wrote: On Thu, 08 Oct 2009 12:10:01 +0200, Robert O'Callahan rob...@ocallahan.org wrote: Another issue is that it's not completely clear to me what is meant by While the user agent might still need network access to obtain parts of the media resourcehttp://www.whatwg.org/specs/web-apps/current-work/#media-resource ... What if there is data in the resource that we don't need in order to play through normally, but which might be needed in some special situations (e.g., enabling subtitles, or seeking using an index), and we optimize to not load that data unless/until we need it? In that case would we never reach NETWORK_LOADED? As I understand it, NETWORK_LOADED means that all bytes of the resource have been loaded, regardless of whether they will be used or not. Are there any formats that would actually allow not downloading parts of the resource in a meaningful way? Yes. A disabled track in an MPEG-4 or QuickTime file is not rendered so the data is not used when presenting the movie. Media data for an enabled but invisible video track (eg. size 0x0, or not within the visible region) or an enabled but muted audio track isn't technically needed for the presentation either. Subtitles and indexes are too small to bother, and multiplexed audio/ video tracks can hardly be skipped without zillions of HTTP Range requests. It seems to me that kind of thing would have to be done either with a server side media fragment request (using the 'track' dimension) or with an external audio/video track somehow synced to the master track (much like external subtitles). I don't agree that this is necessarily best done on a server. Some file formats include tables with the location of every sample, so a media engine that uses range requests anyway can easily read just the data needed. It might be wise for such an engine to optimize the size of chunks read from the server, but that is an implementation detail. Also remember that multiplexed is a relative term, different chunking/interleaving schemes make sense for different media types and use cases so not all multiplexed files interleave data in small chunks. In general NETWORK_LOADED and the load event seem rather useless and dangerous IMHO. If you're playing a resource that doesn't fit in your cache then you'll certainly never reach NETWORK_LOADED, and since authors can't know the cache size they can never rely on load firing. And if you allow the cache discarding behavior I described above, authors can't rely on data actually being present locally even after load has fired. If data can be evicted from the cache you can never reach NETWORK_LOADED because Network connectivity could be lost without affecting the media playback. I suspect many authors will make invalid assumptions about load being sure to fire and about what load means if it does fire. Does anyone have any use cases that load actually solves? I also agree that the 'load' event and the NETWORK_LOADED state are not terribly useful and will likely cause a great deal of confusion for developers. We have have seen a number of cases where experienced web developers have used the 'load' event when they should have used the 'canplaythough', and I fear that this will be a common mistake. I agree, sites that depend on the load event sites will likely break randomly for file sizes that usually barely fit into the cache of the browser they were tested with. If browsers are conservative with bandwidth and only send the load event when it's true, I think we will have less of a problem however. I don't agree that it will be any less of a problem if browsers are conservative, users will still not *ever* be able to depend on the 'load' event firing (except perhaps for local files). Note that the load event isn't strictly needed, waiting for a progress event with loaded==total would achieve the same thing. Actually, a progress event with loaded==total tells you even less than the 'load' event because it doesn't guarantee that the data won't be evicted from the cache. Aesthetically, however, I think it would be strange to not have the load event. I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it (including many of the WebKit layout tests :-( ), but I think we will be better off in the long run if we get rid of the event and network state now. eric
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Thu, 08 Oct 2009 19:42:14 +0200, Eric Carlson eric.carl...@apple.com wrote: On Oct 8, 2009, at 5:32 AM, Philip Jägenstedt wrote: On Thu, 08 Oct 2009 12:10:01 +0200, Robert O'Callahan rob...@ocallahan.org wrote: Another issue is that it's not completely clear to me what is meant by While the user agent might still need network access to obtain parts of the media resourcehttp://www.whatwg.org/specs/web-apps/current-work/#media-resource ... What if there is data in the resource that we don't need in order to play through normally, but which might be needed in some special situations (e.g., enabling subtitles, or seeking using an index), and we optimize to not load that data unless/until we need it? In that case would we never reach NETWORK_LOADED? As I understand it, NETWORK_LOADED means that all bytes of the resource have been loaded, regardless of whether they will be used or not. Are there any formats that would actually allow not downloading parts of the resource in a meaningful way? Yes. A disabled track in an MPEG-4 or QuickTime file is not rendered so the data is not used when presenting the movie. Media data for an enabled but invisible video track (eg. size 0x0, or not within the visible region) or an enabled but muted audio track isn't technically needed for the presentation either. The question was rather of such tracks can be multiplexed in a way that make it possible to skip them efficiently. As per below, probably yes. Subtitles and indexes are too small to bother, and multiplexed audio/ video tracks can hardly be skipped without zillions of HTTP Range requests. It seems to me that kind of thing would have to be done either with a server side media fragment request (using the 'track' dimension) or with an external audio/video track somehow synced to the master track (much like external subtitles). I don't agree that this is necessarily best done on a server. Some file formats include tables with the location of every sample, so a media engine that uses range requests anyway can easily read just the data needed. It might be wise for such an engine to optimize the size of chunks read from the server, but that is an implementation detail. Also remember that multiplexed is a relative term, different chunking/interleaving schemes make sense for different media types and use cases so not all multiplexed files interleave data in small chunks. Right, if the chunks are large enough (at least 100s of KB I suppose) it might be worthwhile using range requests. In general NETWORK_LOADED and the load event seem rather useless and dangerous IMHO. If you're playing a resource that doesn't fit in your cache then you'll certainly never reach NETWORK_LOADED, and since authors can't know the cache size they can never rely on load firing. And if you allow the cache discarding behavior I described above, authors can't rely on data actually being present locally even after load has fired. If data can be evicted from the cache you can never reach NETWORK_LOADED because Network connectivity could be lost without affecting the media playback. I suspect many authors will make invalid assumptions about load being sure to fire and about what load means if it does fire. Does anyone have any use cases that load actually solves? I also agree that the 'load' event and the NETWORK_LOADED state are not terribly useful and will likely cause a great deal of confusion for developers. We have have seen a number of cases where experienced web developers have used the 'load' event when they should have used the 'canplaythough', and I fear that this will be a common mistake. I agree, sites that depend on the load event sites will likely break randomly for file sizes that usually barely fit into the cache of the browser they were tested with. If browsers are conservative with bandwidth and only send the load event when it's true, I think we will have less of a problem however. I don't agree that it will be any less of a problem if browsers are conservative, users will still not *ever* be able to depend on the 'load' event firing (except perhaps for local files). Precisely, if browsers don't fire 'load' until the last data is loaded when playback is closing in on completion, people can't accidentally use it instead of 'canplay'. Still, for local files, very small files or in combination with autobuffer the problem will still be the same. Note that the load event isn't strictly needed, waiting for a progress event with loaded==total would achieve the same thing. Actually, a progress event with loaded==total tells you even less than the 'load' event because it doesn't guarantee that the data won't be evicted from the cache. Good point. Aesthetically, however, I think it would be strange to not have the load event. I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, Oct 9, 2009 at 7:31 AM, Philip Jägenstedt phil...@opera.com wrote: I wouldn't be particularly opposed to dropping the load event, unless there's a use case for the guarantee that the resource won't be evicted from cache as long as the page is loaded. If there is a need for that, then I think we should add some kind of pinning API, although I'm not sure how it would avoid the problem of greedy authors pinning resources unnecessarily. But we'd have to consider the actual use cases. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, Oct 9, 2009 at 1:32 AM, Philip Jägenstedt phil...@opera.com wrote: The spec notes that Some resources, e.g. streaming Web radio, can never reach the NETWORK_LOADED state. In my understanding, you mustn't go to NETWORK_LOADED if you can't guarantee that the resource will remain in cache. Browsers with clever caching or small caches simply won't send a load event most of the time. Right, HTML5 allows Gecko to simply never enter the NETWORK_LOADED state and send a load event. That would be the simplest thing for us to do, and I think the best in terms of letting us do intelligent caching. But I'd prefer not to do it alone. Aesthetically, however, I think it would be strange to not have the load event. If you mean because of the analogy with the load events of other elements, then I strongly disagree. That analogy is itself the dangerous thing about media element load events: the name strongly suggests you can use them the same way as the load events for, say, documents, whereas in reality you can't, in particular because you cannot rely on the load event eventually firing. (The fact that the load event will usually fire during testing makes it even more dangerous.) Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, Oct 9, 2009 at 6:42 AM, Eric Carlson eric.carl...@apple.com wrote: I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it (including many of the WebKit layout tests :-( ), but I think we will be better off in the long run if we get rid of the event and network state now. Me too. I'm game if you are! So I propose: 1) Remove the NETWORK_LOADED state and load and loadend events from media elements. The resource fetch algorithm simply never transitions from step 2 to step 3. 2) Modify Gecko and Webkit accordingly. If we do part 2, which I think is already permitted by the spec, then authors will stop depending on load whether or not we get consensus for altering the spec. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] Restarting the media element resource fetch algorithm after load event
On Fri, Oct 9, 2009 at 8:19 AM, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Oct 9, 2009 at 6:42 AM, Eric Carlson eric.carl...@apple.com wrote: I am not worried about the aesthetics of not having the event. I am somewhat concerned about existing content that uses it (including many of the WebKit layout tests :-( ), but I think we will be better off in the long run if we get rid of the event and network state now. Me too. I'm game if you are! So I propose: 1) Remove the NETWORK_LOADED state and load and loadend events from media elements. The resource fetch algorithm simply never transitions from step 2 to step 3. 2) Modify Gecko and Webkit accordingly. If we do part 2, which I think is already permitted by the spec, then authors will stop depending on load whether or not we get consensus for altering the spec. I think what people are really after when they look for the load or loadend events is that they can do something with the media element - which is really what the loadedmetadata event does. So, I think we're safe with removing the load and loadend events as well as the NETWORK_LOADED state, since I can't really think of a situation where those are important to the developers. As long as the browser has access to the complete data and is capable of doing something with it - such as any kind of video data analysis - it doesn't really matter whether it's in the local cache or on the server or still in transit. I agree, we're safer removing the surplus events and state. Cheers, Silvia.
