Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Sep 28, 2016 at 5:18 PM, Garrett Smith  wrote:
> On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
>> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>>
>>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>>> time-stretching (which is speed change with pitch compensation).
>>
>> That's neat. If you're interested in exposing this as an API on
>> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> might work?
>>
>
> Please post a link to that spec bug.

I don't think a bug was ever filed, but you can file one at
https://github.com/whatwg/html/issues

-- 
Philip Jägenstedt

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>
>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>> time-stretching (which is speed change with pitch compensation).
>
> That's neat. If you're interested in exposing this as an API on
> HTMLMediaElement, can you file a spec bug with a proposal of how it
> might work?
>

Please post a link to that spec bug.

Thanks,
-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

All,

There is no way to independently adjust pitch and speed of HTML5 video.
(Now, Chrome doesn't support preservesPitch (in any prefix), so things are
worse.)

Generally it is a good idea to read all of the thread's messages before
replying.

Garrett: Which implementations does this work in?
Jer: None, because there is no such thing as a PitchShiftNode.

Further discussion of the proposed API followed with comments of Jer's
proposed API being a lot work to do something simple.

Domenic should have read that, just like he should have read (and heeded)
the warning in my response to his harassing personal emails, to not email
me anymore, ever again.

Thank you,

On Wed, Jul 20, 2016 at 9:48 PM, Domenic Denicola  wrote:

> From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of
> Garrett Smith
>
> > What is the status on this?
>
> I believe the conclusion of the thread was that this is possible with the
> Web Audio API.
>
>


-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Garrett Smith

> What is the status on this?

I believe the conclusion of the thread was that this is possible with the Web 
Audio API.

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

What is the status on this?

Need to be able to independently adjust pitch and slow down playback.
I need this on a daily basis.

In Chrome pitch is now preserved regardless of setting
vid.[prefix]preservesPitch = false.



On Mon, Mar 14, 2016 at 1:13 AM, Xabier Rodríguez Calvar
 wrote:
> Hi,
>
> O Ven, 11-03-2016 ás 13:19 -0800, Jer Noble escribiu:
>> > Oh. Poor hardware integration, and now this…. Being Apple CEO is
>> not Tim Cook's greatest gift…
>>
>> I’m not sure how that’s relevant.
>
> Yes I agree with Jer here, it looks like a very bad, undesired and
> irrelevant joke.
>
> Best regards.



-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 11, 2016 at 1:19 PM, Jer Noble  wrote:
>
> On Mar 11, 2016, at 1:11 PM, Garrett Smith  wrote:
>

…

> I also found:—
> https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVAudioUnitTimePitch_Class/
> ("preliminary document for an API or technology in development.”)
>
>
> You’ll be interested to know that the design of the Web Audio API was
> heavily influenced by the design of the OS X Core Audio library, and that is
> class you’ve found is an Objective-C wrapper around a Core Audio node.  So
> yes, this class could well be used to implement a Web Audio PitchShift node.
>
> -Jer

If a PitchShift Node can do the job, then great! But, the API should
be much more straightforward, such as with an Adapter, so we have
playbackRate and playbackSpeed, and not a long string of a.b.c().d …

On the VIDEO — separate sliders for pitch and for playback rate. One
slider for speed (playback rate) and another for pitch. The sliders
MUST NOT affect each others' behavior. Imagine you're using the video
player and the speed slider affects the pitch, but only after you've
used the pitch slider affect. Obviously no good. And that is why it
doesn't exist.


Thank you,
-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Xabier Rodríguez Calvar]

Hi,

O Ven, 11-03-2016 ás 13:19 -0800, Jer Noble escribiu:
> > Oh. Poor hardware integration, and now this…. Being Apple CEO is
> not Tim Cook's greatest gift… 
> 
> I’m not sure how that’s relevant.

Yes I agree with Jer here, it looks like a very bad, undesired and
irrelevant joke.

Best regards.

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 11, 2016, at 1:11 PM, Garrett Smith  wrote:
> 
> 
> 
> On Tuesday, March 8, 2016, Jer Noble  > wrote:
> 
> > On Mar 8, 2016, at 4:42 PM, Garrett Smith > wrote:
> >
> > On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble > wrote:
> >>
> >>> On Mar 4, 2016, at 3:19 PM, Garrett Smith > 
> >>> wrote:
> >>>
> >>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble > wrote:
> 
> > On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt > 
> > wrote:
> >
> > On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  > <>> wrote:
> >> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  >> <>> wrote:
> >>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> >>> >
> >>> wrote:
>  On 11/12/15, Philip Jägenstedt > wrote:
> > On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> > >
> > wrote:
> >>
> >> On 10/19/15, Philip Jägenstedt > wrote:
> >>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> >>> >
> >>> wrote:
>  On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>  >
>  wrote:
> > From: Eric Carlson [mailto:eric.carl...@apple.com <>]
> 
>  
> >>
> >> The Web Audio equivalent would be:
> >>
> >> var video = document.querySelector(‘video’);
> >> video.preservesPitch = false;
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> >
> > Which implementations does that work in?
> 
> None, because there is no such thing as a PitchShiftNode.
> 
> 
> I see. 
>  
> > That code is more complex than should be necessary. I see where you're
> > coming from on separating the audio. Could we move the media decorator
> > behind the scenes, and replace it with a simple getter/setter property
> > like `videoElement.audio` so that that can happen automagically?
> > Reminds me of createElement, createRange, document.implementation,
> > etc. Warts!
> 
> I’m not entirely sure what you’re asking here. If it’s that you don’t like 
> the `context.createMediaElementSource()` or `context.createPitchShift()` 
> syntax and would rather a constructor syntax, Issue #250 
>  > in the Web Audio spec 
> is the issue for you.
> 
> > But then again, you also just said that there are no APIs on OS X that
> > allow an arbitrary pitch shift to be added to audio. If that is true,
> > then your `createPitchShift` code would be possible anyway, is that
> 
> There is no such API for such post-processing built into the OS X media 
> frameworks. 
> 
> Oh. Poor hardware integration, and now this…. Being Apple CEO is not Tim 
> Cook's greatest gift… 

I’m not sure how that’s relevant.

> As an example, there is an API for preserving pitch across rate changes: 
> -[AVPlayerItem setAudioTimePitchAlgorithm:] 
>   
> >.
>   So implementing the HTMLMediaElement “preservesPitch" property is trivial.  
> There is no such API for applying a specific and arbitrary amount of pitch 
> shifting. It would have to be implemented entirely by hand, the same way we 
> implement processing of media element audio with the Web Audio API.  Which 
> gets back to: at this point, we should just implement a Pitch Shift node in 
> Web Audio.
> 
> 
>  I see.
> 
> >> var video = document.querySelector(‘video’);
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> 
> Too many steps and implementation details. It's complicated. It's asking for 
> more js libraries and wrappers for that, and other situations related to 
> videos' audio.  Make it simpler. Hide the implementation details.
> 
> Hide:—
>  * AudioContext creation and subsequent video connection.
>  * createPitchShift — use audio.pitchShift to get a pitchShiftNode
>  * pitchShiftNode.shiftAmount — 

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Tuesday, March 8, 2016, Jer Noble  wrote:

>
> > On Mar 8, 2016, at 4:42 PM, Garrett Smith 
> wrote:
> >
> > On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble  wrote:
> >>
> >>> On Mar 4, 2016, at 3:19 PM, Garrett Smith 
> wrote:
> >>>
> >>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
> 
> > On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt 
> wrote:
> >
> > On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith <
> dhtmlkitc...@gmail.com> wrote:
> >> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt <
> phil...@opera.com> wrote:
> >>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
> >>> wrote:
>  On 11/12/15, Philip Jägenstedt  wrote:
> > On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
> > wrote:
> >>
> >> On 10/19/15, Philip Jägenstedt  wrote:
> >>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <
> phil...@opera.com>
> >>> wrote:
>  On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola <
> d...@domenic.me>
>  wrote:
> > From: Eric Carlson [mailto:eric.carl...@apple.com]


>
>
>>
> >> The Web Audio equivalent would be:
> >>
> >> var video = document.querySelector(‘video’);
> >> video.preservesPitch = false;
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> >
> > Which implementations does that work in?
>
> None, because there is no such thing as a PitchShiftNode.
>
>
I see.


> > That code is more complex than should be necessary. I see where you're
> > coming from on separating the audio. Could we move the media decorator
> > behind the scenes, and replace it with a simple getter/setter property
> > like `videoElement.audio` so that that can happen automagically?
> > Reminds me of createElement, createRange, document.implementation,
> > etc. Warts!
>
> I’m not entirely sure what you’re asking here. If it’s that you don’t like
> the `context.createMediaElementSource()` or `context.createPitchShift()`
> syntax and would rather a constructor syntax, Issue #250 <
> https://github.com/WebAudio/web-audio-api/issues/250> in the Web Audio
> spec is the issue for you.
>
> > But then again, you also just said that there are no APIs on OS X that
> > allow an arbitrary pitch shift to be added to audio. If that is true,
> > then your `createPitchShift` code would be possible anyway, is that
>
> There is no such API for such post-processing built into the OS X media
> frameworks.


Oh. Poor hardware integration, and now this…. Being Apple CEO is not Tim
Cook's greatest gift…

As an example, there is an API for preserving pitch across rate changes:
> -[AVPlayerItem setAudioTimePitchAlgorithm:] <
> https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVPlayerItem_Class/index.html#//apple_ref/occ/instp/AVPlayerItem/audioTimePitchAlgorithm>.
> So implementing the HTMLMediaElement “preservesPitch" property is trivial.
> There is no such API for applying a specific and arbitrary amount of pitch
> shifting. It would have to be implemented entirely by hand, the same way we
> implement processing of media element audio with the Web Audio API.  Which
> gets back to: at this point, we should just implement a Pitch Shift node in
> Web Audio.
>
>
 I see.

>> var video = document.querySelector(‘video’);
>> var context = new AudioContext();
>> var sourceNode = context.createMediaElementSource(video);
>> var pitchShiftNode = context.createPitchShift();
>> pitchShiftNode.shiftAmount = 1.023;
>> sourceNode.connect(pitchShiftNode);
>> pitchShiftNode.connect(context.destination);
>>

Too many steps and implementation details. It's complicated. It's asking
for more js libraries and wrappers for that, and other situations related
to videos' audio.  Make it simpler. Hide the implementation details.

Hide:—
 * AudioContext creation and subsequent video connection.
 * createPitchShift — use audio.pitchShift to get a pitchShiftNode
 * pitchShiftNode.shiftAmount — make pitchShiftNode a getter/setter (like
video.playbackRate)

>> var video = document.querySelector(‘video’);
>> var context = new AudioContext();
>> var sourceNode = context.createMediaElementSource(video);
>> var context = new AudioContext();

// Hidden AudioContext construction
var audio = video.audio;

>> var pitchShiftNode = context.createPitchShift();
>> pitchShiftNode.shiftAmount = 1.023;

//  Hidden AudioContext createPitchShift factory.
//  Setter on pitchShift
audio.pitchShift = 1.023;

>> sourceNode.connect(pitchShiftNode);
>> 

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble  wrote:
>
>> On Mar 4, 2016, at 3:19 PM, Garrett Smith  wrote:
>>
>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>>>
 On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:

 On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  
 wrote:
> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
> wrote:
>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>> wrote:
>>> On 11/12/15, Philip Jägenstedt  wrote:
 On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
 wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>> 
>> wrote:
>>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>>> wrote:
 From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>> Two things.
>>
>> 1. Do the underlying media frameworks that browsers are using support
>> arbitrary pitch changes, or do they also only have the limited
>> preservesPitch-style API?
>
> Are there any problems getting in the way of pitch adjustment (without
> depending on playbackRate)?

 I don't know, that was basically my question too. If the underlying
 APIs don't support it, that's a problem that needs to be fixed first.
>>>
>>>
>>> There are no such APIs on OS X which would allow an arbitrary pitch shift 
>>> to be added to an otherwise normally playing piece of audio.
>>>
>>> IMO, this is a more appropriate request for the Web Audio API (adding a 
>>> Audio Node which can add an arbitrary amount of pitch shift).  At which 
>>> point, there would be no need for this in HTMLMediaElement, as authors 
>>> could make a simple node graph consisting of an MediaElementAudioSourceNode 
>>> and a PitchShiftNode.
>>>
>>> -Jer
>>
>> But that can't work on OSX, right?  I wonder how audio software on mac
>> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
>> many others can all do this.
>
> None of them use the built in platform APIs to shift the pitch of encoded 
> media.  Each does it manually, within their app, and each probably uses a 
> different algorithm to achieve shift in pitch.
>
>> Plus how would web Audio API solve for the use case?
>>
>> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
>> song is about a 50 cents flat of standard tuning. The pitch can be
>> adjusted by setting playbackRate to 1.023 and setting
>> MozPreservesPitch to false:—
>>
>> var vv = document.querySelector("video");
>> vv.mozPreservesPitch = 0;
>> vv.playbackRate = 1.023
>>
>> — but that speeds it up. I don't want speed coupled with pitch.
>
> The Web Audio equivalent would be:
>
> var video = document.querySelector(‘video’);
> video.preservesPitch = false;
> var context = new AudioContext();
> var sourceNode = context.createMediaElementSource(video);
> var pitchShiftNode = context.createPitchShift();
> pitchShiftNode.shiftAmount = 1.023;
> sourceNode.connect(pitchShiftNode);
> pitchShiftNode.connect(context.destination);
>

Which implementations does that work in?

That code is more complex than should be necessary. I see where you're
coming from on separating the audio. Could we move the media decorator
behind the scenes, and replace it with a simple getter/setter property
like `videoElement.audio` so that that can happen automagically?
Reminds me of createElement, createRange, document.implementation,
etc. Warts!

But then again, you also just said that there are no APIs on OS X that
allow an arbitrary pitch shift to be added to audio. If that is true,
then your `createPitchShift` code would be possible anyway, is that
right?

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 4, 2016, at 3:19 PM, Garrett Smith  wrote:
> 
> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>> 
>>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
>>> 
>>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  
>>> wrote:
 On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
 wrote:
> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> wrote:
>> On 11/12/15, Philip Jägenstedt  wrote:
>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>>> wrote:
 
 On 10/19/15, Philip Jägenstedt  wrote:
> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> wrote:
>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>> wrote:
>>> From: Eric Carlson [mailto:eric.carl...@apple.com]
 
> Two things.
> 
> 1. Do the underlying media frameworks that browsers are using support
> arbitrary pitch changes, or do they also only have the limited
> preservesPitch-style API?
 
 Are there any problems getting in the way of pitch adjustment (without
 depending on playbackRate)?
>>> 
>>> I don't know, that was basically my question too. If the underlying
>>> APIs don't support it, that's a problem that needs to be fixed first.
>> 
>> 
>> There are no such APIs on OS X which would allow an arbitrary pitch shift to 
>> be added to an otherwise normally playing piece of audio.
>> 
>> IMO, this is a more appropriate request for the Web Audio API (adding a 
>> Audio Node which can add an arbitrary amount of pitch shift).  At which 
>> point, there would be no need for this in HTMLMediaElement, as authors could 
>> make a simple node graph consisting of an MediaElementAudioSourceNode and a 
>> PitchShiftNode.
>> 
>> -Jer
> 
> But that can't work on OSX, right?  I wonder how audio software on mac
> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
> many others can all do this.

None of them use the built in platform APIs to shift the pitch of encoded 
media.  Each does it manually, within their app, and each probably uses a 
different algorithm to achieve shift in pitch.

> Plus how would web Audio API solve for the use case?
> 
> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
> song is about a 50 cents flat of standard tuning. The pitch can be
> adjusted by setting playbackRate to 1.023 and setting
> MozPreservesPitch to false:—
> 
> var vv = document.querySelector("video");
> vv.mozPreservesPitch = 0;
> vv.playbackRate = 1.023
> 
> — but that speeds it up. I don't want speed coupled with pitch.

The Web Audio equivalent would be:

var video = document.querySelector(‘video’);
video.preservesPitch = false;
var context = new AudioContext();
var sourceNode = context.createMediaElementSource(video);
var pitchShiftNode = context.createPitchShift();
pitchShiftNode.shiftAmount = 1.023;
sourceNode.connect(pitchShiftNode);
pitchShiftNode.connect(context.destination);

-Jer

> Thanks,
> -- 
> Garrett
> @xkit
> ChordCycles.wordpress.com
> garretts.github.io
> personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>
>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
>>
>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
>>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
>>> wrote:
 On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
 wrote:
> On 11/12/15, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>> wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
 On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>>
 Two things.

 1. Do the underlying media frameworks that browsers are using support
 arbitrary pitch changes, or do they also only have the limited
 preservesPitch-style API?
>>>
>>> Are there any problems getting in the way of pitch adjustment (without
>>> depending on playbackRate)?
>>
>> I don't know, that was basically my question too. If the underlying
>> APIs don't support it, that's a problem that needs to be fixed first.
>
>
> There are no such APIs on OS X which would allow an arbitrary pitch shift to 
> be added to an otherwise normally playing piece of audio.
>
> IMO, this is a more appropriate request for the Web Audio API (adding a Audio 
> Node which can add an arbitrary amount of pitch shift).  At which point, 
> there would be no need for this in HTMLMediaElement, as authors could make a 
> simple node graph consisting of an MediaElementAudioSourceNode and a 
> PitchShiftNode.
>
> -Jer

But that can't work on OSX, right?  I wonder how audio software on mac
does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
many others can all do this.

Plus how would web Audio API solve for the use case?

To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
song is about a 50 cents flat of standard tuning. The pitch can be
adjusted by setting playbackRate to 1.023 and setting
MozPreservesPitch to false:—

var vv = document.querySelector("video");
vv.mozPreservesPitch = 0;
vv.playbackRate = 1.023

— but that speeds it up. I don't want speed coupled with pitch.

Thanks,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
> 
> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
>> wrote:
>>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>>> wrote:
 On 11/12/15, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> wrote:
>> 
>> On 10/19/15, Philip Jägenstedt  wrote:
>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> wrote:
 On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
 wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
>> 
>>> Two things.
>>> 
>>> 1. Do the underlying media frameworks that browsers are using support
>>> arbitrary pitch changes, or do they also only have the limited
>>> preservesPitch-style API?
>> 
>> Are there any problems getting in the way of pitch adjustment (without
>> depending on playbackRate)?
> 
> I don't know, that was basically my question too. If the underlying
> APIs don't support it, that's a problem that needs to be fixed first.


There are no such APIs on OS X which would allow an arbitrary pitch shift to be 
added to an otherwise normally playing piece of audio.

IMO, this is a more appropriate request for the Web Audio API (adding a Audio 
Node which can add an arbitrary amount of pitch shift).  At which point, there 
would be no need for this in HTMLMediaElement, as authors could make a simple 
node graph consisting of an MediaElementAudioSourceNode and a PitchShiftNode.

-Jer

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Thu, Mar 3, 2016 at 1:18 AM, Paul Adenot  wrote:
> That looks appropriate, and is in line with how the Web Audio API does
> detuning (using the same unit, [0]).
>
> That said, I'm not sure about the use case here? Time-stretching (normal
> playbackRate) is useful, for example, to watch a conference slightly faster
> in order to save some time. I can't really find non-musical use-cases, and
> musical use-cases are maybe better served using the Web Audio API.
>

In summary:

User wants to hear the audio at the desired pitch, for accompaniment
or learning purposes, ideally while simultaneously watching the video
at the desired speed, using a video sharing or lesson websites.

User finds a version of the artist performing the piece among other
versions and lessons in various tunings. The user uses adjusts video's
pitch, independently of playback speed, to match his instrument's
tuning using.

The pitch control can be provided by a user script, bookmarklet, or extension.*

Existing software for this includes Amazing Slow Downer.

(more below)

Thank you,


> Paul.
>
> [0]:
> http://webaudio.github.io/web-audio-api/#widl-AudioBufferSourceNode-detune
>
> On Thu, Mar 3, 2016 at 12:00 AM, Garrett Smith 
> wrote:
>>
>> On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith 
>> wrote:
>> > On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt 
>> > wrote:
>> >> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot 
>> >> wrote:
>> >>>
>> >>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to
>> >>> time-stretching (which is speed change with pitch compensation).
>> >>
>> >> That's neat. If you're interested in exposing this as an API on
>> >> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> >> might work?
>> >>
>> >
>> > Here is what I had in mind; hope it helps.
>> >
>> > Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
>> > pitch in cents.
>> >
>> > Equally tempered semitones span 100 cents.
>> >
>> > The developer can use INPUT type="range" with an oninput, to allow the
>> > user to adjust the video's pitch, in cents.
>> >
>> > interface HTMLMediaElement : HTMLElement {
>> >  attribute unsigned short pitchAdjustment;
>> >  …
>> > }
>> >
>>
>> Not unsigned, sorry.
>>
>> interface HTMLMediaElement : HTMLElement {
>>   attribute short pitchAdjustment;
>>   …
>>  }
>>
>> That should allow for pitch adjustment over 54 octaves. This is
>> withing human hearing range of 10 octaves. 1 having the value of one
>> cent, gives 1200 cents per octave.
>>
>> let
>>  semitone = 100, // cents
>>  octave = 12 * semitone, // 1200
>>  sizeOfShort = 65535;
>> sizeOfShort / octave 54.6125
>>
>> Signed short range is −32768 to 32767
>> --
>> Garrett
>> @xkit
>> ChordCycles.wordpress.com
>> garretts.github.io
>> personx.tumblr.com
>
>



-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Paul Adenot]

That looks appropriate, and is in line with how the Web Audio API does
detuning (using the same unit, [0]).

That said, I'm not sure about the use case here? Time-stretching (normal
playbackRate) is useful, for example, to watch a conference slightly faster
in order to save some time. I can't really find non-musical use-cases, and
musical use-cases are maybe better served using the Web Audio API.

Paul.

[0]:
http://webaudio.github.io/web-audio-api/#widl-AudioBufferSourceNode-detune

On Thu, Mar 3, 2016 at 12:00 AM, Garrett Smith 
wrote:

> On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith 
> wrote:
> > On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt 
> wrote:
> >> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot 
> wrote:
> >>>
> >>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to
> time-stretching (which is speed change with pitch compensation).
> >>
> >> That's neat. If you're interested in exposing this as an API on
> >> HTMLMediaElement, can you file a spec bug with a proposal of how it
> >> might work?
> >>
> >
> > Here is what I had in mind; hope it helps.
> >
> > Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
> > pitch in cents.
> >
> > Equally tempered semitones span 100 cents.
> >
> > The developer can use INPUT type="range" with an oninput, to allow the
> > user to adjust the video's pitch, in cents.
> >
> > interface HTMLMediaElement : HTMLElement {
> >  attribute unsigned short pitchAdjustment;
> >  …
> > }
> >
>
> Not unsigned, sorry.
>
> interface HTMLMediaElement : HTMLElement {
>   attribute short pitchAdjustment;
>   …
>  }
>
> That should allow for pitch adjustment over 54 octaves. This is
> withing human hearing range of 10 octaves. 1 having the value of one
> cent, gives 1200 cents per octave.
>
> let
>  semitone = 100, // cents
>  octave = 12 * semitone, // 1200
>  sizeOfShort = 65535;
> sizeOfShort / octave 54.6125
>
> Signed short range is −32768 to 32767
> --
> Garrett
> @xkit
> ChordCycles.wordpress.com
> garretts.github.io
> personx.tumblr.com
>

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith  wrote:
> On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
>> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>>
>>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>>> time-stretching (which is speed change with pitch compensation).
>>
>> That's neat. If you're interested in exposing this as an API on
>> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> might work?
>>
>
> Here is what I had in mind; hope it helps.
>
> Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
> pitch in cents.
>
> Equally tempered semitones span 100 cents.
>
> The developer can use INPUT type="range" with an oninput, to allow the
> user to adjust the video's pitch, in cents.
>
> interface HTMLMediaElement : HTMLElement {
>  attribute unsigned short pitchAdjustment;
>  …
> }
>

Not unsigned, sorry.

interface HTMLMediaElement : HTMLElement {
  attribute short pitchAdjustment;
  …
 }

That should allow for pitch adjustment over 54 octaves. This is
withing human hearing range of 10 octaves. 1 having the value of one
cent, gives 1200 cents per octave.

let
 semitone = 100, // cents
 octave = 12 * semitone, // 1200
 sizeOfShort = 65535;
sizeOfShort / octave 54.6125

Signed short range is −32768 to 32767
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>
>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>> time-stretching (which is speed change with pitch compensation).
>
> That's neat. If you're interested in exposing this as an API on
> HTMLMediaElement, can you file a spec bug with a proposal of how it
> might work?
>

Here is what I had in mind; hope it helps.

Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
pitch in cents.

Equally tempered semitones span 100 cents.

The developer can use INPUT type="range" with an oninput, to allow the
user to adjust the video's pitch, in cents.

interface HTMLMediaElement : HTMLElement {
 attribute unsigned short pitchAdjustment;
 …
}

pitchAdjustment adjusts the target HTMLMediaElement's audio pitch
relative to its original pitch, in cents. A value of 0 represents no
pitch adjustment.

Getting gets the last set value; 0 if unset. Setting sets the value
and changes the pitch in cents (relative to the media element's
original pitch).

Example:

http://example.net/; id="vv"">




function updateMoviePitch(ev) {
  var vv = document.getElementById("vv");
  vv.pitchAdjustment = +this.value;
  showPitchUI(vv, this.value);
}

Thank you,
-- 
Garrett
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>
> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
> time-stretching (which is speed change with pitch compensation).

That's neat. If you're interested in exposing this as an API on
HTMLMediaElement, can you file a spec bug with a proposal of how it
might work?

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Paul Adenot]

Gecko uses code that can do arbitrary pitch adjustment, unrelated to
time-stretching (which is speed change with pitch compensation).

Paul.

On Wed, Mar 2, 2016 at 5:00 AM, Philip Jägenstedt  wrote:

> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith 
> wrote:
> > On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt 
> wrote:
> >> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith  >
> >> wrote:
> >>> On 11/12/15, Philip Jägenstedt  wrote:
>  On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
>  wrote:
> >
> > On 10/19/15, Philip Jägenstedt  wrote:
> > > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <
> phil...@opera.com>
> > > wrote:
> > >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> > >> wrote:
> > >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
> >
> >> Two things.
> >>
> >> 1. Do the underlying media frameworks that browsers are using support
> >> arbitrary pitch changes, or do they also only have the limited
> >> preservesPitch-style API?
> >
> > Are there any problems getting in the way of pitch adjustment (without
> > depending on playbackRate)?
>
> I don't know, that was basically my question too. If the underlying
> APIs don't support it, that's a problem that needs to be fixed first.
>
> Philip
>

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>> wrote:
>>> On 11/12/15, Philip Jägenstedt  wrote:
 On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
 wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> > wrote:
> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> >> wrote:
> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>> Two things.
>>
>> 1. Do the underlying media frameworks that browsers are using support
>> arbitrary pitch changes, or do they also only have the limited
>> preservesPitch-style API?
>
> Are there any problems getting in the way of pitch adjustment (without
> depending on playbackRate)?

I don't know, that was basically my question too. If the underlying
APIs don't support it, that's a problem that needs to be fixed first.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> wrote:
>> On 11/12/15, Philip Jägenstedt  wrote:
>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>>> wrote:

 On 10/19/15, Philip Jägenstedt  wrote:
 > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 > wrote:
 >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
 >> wrote:
 >>> From: Eric Carlson [mailto:eric.carl...@apple.com]

> Two things.
>
> 1. Do the underlying media frameworks that browsers are using support
> arbitrary pitch changes, or do they also only have the limited
> preservesPitch-style API?

Are there any problems getting in the way of pitch adjustment (without
depending on playbackRate)?

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:43 AM, Eric Carlson  wrote:
>
>> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
>>
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith > > wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
 On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>
>>>
>>> [...]
 I've filed a spec issue to make it so:
 https://github.com/whatwg/html/issues/262

 If there's any implementor interest in pitch control that goes beyond
 (independently) or that, please file a separate issue.

>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>   I am actually quite ambivalent about this feature - it is currently broken 
> on OSX and it has never been implemented on iOS, but as far as I can see we 
> haven’t received a single bug report about this.

Ah. I removed webkitPreservesPitched entirely from Blink because it
wasn't implemented on any platforms, maybe you could do the same on
iOS?

More importantly, is it possible to support this on iOS, and is it
likely to bubble to the top of your priorities any time soon if it
were added to the spec? I don't think it's high priority for anyone
working on Chromium, but unless Safari and Firefox are ready to drop
their prefixed APIs entirely it still seems like a good idea to get
what little interop we can by standardizing.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
wrote:
> On 11/12/15, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>> wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
>>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> > wrote:
>>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>>> >> wrote:
>>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>> >>>
>>>
>>> [...]
>>> > I've filed a spec issue to make it so:
>>> > https://github.com/whatwg/html/issues/262
>>> >
>>> > If there's any implementor interest in pitch control that goes beyond
>>> > (independently) or that, please file a separate issue.
>>> >
>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>
> https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

I don't understand, isn't the AudioTrack API (enabling/disabling individual
audio tracks) separate a separate concern? Or are you saying that there's
no way to get the audio tracks separately into the Web Audio API in order
to process them differently? That is certainly true, and if having that
capability would allow you to solve the problem with the Web Audio API, I
think that seems like a powerful primitive to have.

>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>
> The situation where you want hear the audio at the desired pitch,
> while simultaneously watching the video at the desired speed, to see
> it performed, and to play along. It doesn't work well when trying to
> play to a video example, such as playing an open G chord, and the
> sound from the video does not match the sound from your own guitar.
>
> What I can do now on websites that use VIDEO, such as YouTube, is
> limited to using the web inspector (console). I click "inspect
> element" to find the VIDEO element, then change its DOM properties for
> playbackRate = .95 and mozPreservesPitch = false. That'll get
> everything down a half-step (although at 95% speed).
>
> For example, watch Andy James play "Harvester of Sorrow" but hear it
> 1/2 step down:
>
> https://www.youtube.com/watch?v=vYZjC_U1VJg
>
> 1) Right click on the video "Inspect Element"
> 2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
> 3) set playbackRate: 0.95, mozPreservesPitch: 0
>
> Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
> few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
> down further.
>
> A=440 (standard tuning) is much more common. When you want to learn
> something played tuned down 1/2 step and your guitar is in standard
> tuning, it is a problem. Constantly retuning the instrument is not
> practical. Especially for guitars with locking nut systems or for
> other instruments like piano. I don't know about learning horn parts.
> Or you can buy multiple instruments and maintain different tunings for
> them.
>
> Fast passages, it's helpful to slow the speed down but keep the pitch
> as original and in some cases, you want to slow the speed down and
> adjust the pitch separately, so you can do things like learning Yngwie
> Malmsteen songs while using a guitar in standard tuning.
>
> I realize there's not much demand for it. But there seems to be
> separation of playback pitch with `preservesPitch` and 

Re: [whatwg] VIDEO and pitchAdjustment

[Eric Carlson]

> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
> 
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  > wrote:
>> 
>> On 10/19/15, Philip Jägenstedt  wrote:
>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> wrote:
 On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
> 
>> 
>> [...]
>>> I've filed a spec issue to make it so:
>>> https://github.com/whatwg/html/issues/262
>>> 
>>> If there's any implementor interest in pitch control that goes beyond
>>> (independently) or that, please file a separate issue.
>>> 
>> 
>> They won't.
>> 
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>> 
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>> 
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>> 
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>> 
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>> 
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
> 
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
> 
> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
> 
  I am actually quite ambivalent about this feature - it is currently broken on 
OSX and it has never been implemented on iOS, but as far as I can see we 
haven’t received a single bug report about this.

eric

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 10/19/15, Philip Jägenstedt  wrote:
> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> wrote:
>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>>

[...]
> I've filed a spec issue to make it so:
> https://github.com/whatwg/html/issues/262
>
> If there's any implementor interest in pitch control that goes beyond
> (independently) or that, please file a separate issue.
>

They won't.

You can hold the standard of "they need to come here and post up
cogent arguments in favor of feature X", but it ain't gonna happen
that way.

There just isn't a whole lot of money in music education. How many
music education companies are W3C members?

Unlike technology companies like Facebook, Google, Nokia, Opera, and
other companies the post here, small music education operations like
artistworks, jammit, licklibrary, etc are more about their domain —
"music" — than they are about technology.

Major music education websites are still using Flash; their developers
are busy fixing broken links, making the login feature, database, etc
work, etc. Flash is not nice but they apparently were not funded or
motivated enough by the existing HTML5 HTMLMediaElement to use it
instead.

Control over playbackRate has a greater value than pitch control. But
those sites don't even allow the students to change the playbackRate
because they're still using Flash.

You won't read posts here about what students have to say about it the
value of having HTML5 vs Flash, or independent control over pitch and
playbackRate.

-- 
Garrett / independent voice
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> > wrote:
> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
> >>>
>
> [...]
> > I've filed a spec issue to make it so:
> > https://github.com/whatwg/html/issues/262
> >
> > If there's any implementor interest in pitch control that goes beyond
> > (independently) or that, please file a separate issue.
> >
>
> They won't.
>
> You can hold the standard of "they need to come here and post up
> cogent arguments in favor of feature X", but it ain't gonna happen
> that way.
>
> There just isn't a whole lot of money in music education. How many
> music education companies are W3C members?
>
> Unlike technology companies like Facebook, Google, Nokia, Opera, and
> other companies the post here, small music education operations like
> artistworks, jammit, licklibrary, etc are more about their domain —
> "music" — than they are about technology.
>
> Major music education websites are still using Flash; their developers
> are busy fixing broken links, making the login feature, database, etc
> work, etc. Flash is not nice but they apparently were not funded or
> motivated enough by the existing HTML5 HTMLMediaElement to use it
> instead.
>
> Control over playbackRate has a greater value than pitch control. But
> those sites don't even allow the students to change the playbackRate
> because they're still using Flash.
>
> You won't read posts here about what students have to say about it the
> value of having HTML5 vs Flash, or independent control over pitch and
> playbackRate.

Have you investigated whether you can achieve your use cases using the
Web Audio API? If it isn't possible, is there a small addition to Web
Audio that would solve the problem?

It is unfortunately quite hard to get all browser vendors (or even
one) interested in implementing support for something that is only
expected to benefit a niche use case, but we should strive to make
available the primitives that would it possible to implement yourself.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 11/12/15, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> wrote:
>>
>> On 10/19/15, Philip Jägenstedt  wrote:
>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>> > wrote:
>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>> >> wrote:
>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>> >>>
>>
>> [...]
>> > I've filed a spec issue to make it so:
>> > https://github.com/whatwg/html/issues/262
>> >
>> > If there's any implementor interest in pitch control that goes beyond
>> > (independently) or that, please file a separate issue.
>> >
>>
>> They won't.
>>
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>>
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>>
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>>
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>>
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>>
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
>
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
>

https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
>

The situation where you want hear the audio at the desired pitch,
while simultaneously watching the video at the desired speed, to see
it performed, and to play along. It doesn't work well when trying to
play to a video example, such as playing an open G chord, and the
sound from the video does not match the sound from your own guitar.

What I can do now on websites that use VIDEO, such as YouTube, is
limited to using the web inspector (console). I click "inspect
element" to find the VIDEO element, then change its DOM properties for
playbackRate = .95 and mozPreservesPitch = false. That'll get
everything down a half-step (although at 95% speed).

For example, watch Andy James play "Harvester of Sorrow" but hear it
1/2 step down:

https://www.youtube.com/watch?v=vYZjC_U1VJg

1) Right click on the video "Inspect Element"
2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
3) set playbackRate: 0.95, mozPreservesPitch: 0

Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
down further.

A=440 (standard tuning) is much more common. When you want to learn
something played tuned down 1/2 step and your guitar is in standard
tuning, it is a problem. Constantly retuning the instrument is not
practical. Especially for guitars with locking nut systems or for
other instruments like piano. I don't know about learning horn parts.
Or you can buy multiple instruments and maintain different tunings for
them.

Fast passages, it's helpful to slow the speed down but keep the pitch
as original and in some cases, you want to slow the speed down and
adjust the pitch separately, so you can do things like learning Yngwie
Malmsteen songs while using a guitar in standard tuning.

I realize there's not much demand for it. But there seems to be
separation of playback pitch with `preservesPitch` and `playbackRate`.
It would be much better with a control to adjustm pitch, if it is not
too much work.

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt  wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>
>>>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>>>
>>> ...
>>>
>>>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
>>> content.
>>
>> That's really helpful, thanks. Combined with Edge's keyframes-only support, 
>> it sounds like we should probably leave the spec as it is.
>>
>> Do you have thoughts on a mozPreservesPitch equivalent?
>
> Should we just standardize HTMLMediaElement.preservesPitch, perhaps?
> Note that WebKit also has webkitPreservesPitch, but I removed it from
> Blink because it didn't actually do anything in Chromium.
>
> In both Gecko and WebKit it defaults to true. Is there anything else
> worth knowing before writing the spec for this?

I've filed a spec issue to make it so:
https://github.com/whatwg/html/issues/262

If there's any implementor interest in pitch control that goes beyond
(independently) or that, please file a separate issue.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/27/15, Robert O'Callahan  wrote:
> On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith 
> wrote:
>

[...]

> But variable pitch control it would be useful for music adjustments
>> like "over the mountain",  "Black Star", "Take your Whiskey Home", all
>> originally recorded at half-step detuning (A=415), none of which match
>> "standard tuning" commonly used in schools, industry, etc. Recordings
>> of Baroque era music often use instruments tuned lower (and also
>> different scale temperment, but that is a different issue). So it
>> would be nice to adjust the pitch with a `pitchAdjustment` property,
>> as a double, to adjust pitch in cents.
>>
>
> I'd prefer to focus on making sure that Web Audio's Audio Workers are
> powerful enough for Web developers to implement this themselves.
>

I hear you, but I don't understand why.

To have control over audio pitch and playback rate, within a video,
with just a simple property, and without having to create a Worker,
would be very powerful for learning music.

Movie displays music notation or tablature, or a live performance. I
have used such.
The user can adjust the pitch, if necessary and change the speed, for learning.

We want to be able to adjust playbackRate and pitchAdjustment
independently. That might hypothetically be possible by directly
typing into the console, using user scripts, or bookmarklets. Online
music schools can build it into their HTML5 players (though today,
most of them still use the Flash player), for normal musicians (who
aren't using the console, etc).
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Michael Enright]

I'm an implementer (to the fullest extent that one can be one by
modifying existing open source), just following the convention of this
list of stating my use case. The use case reflects reality since it
was implemented last year.

On Mon, Aug 31, 2015 at 12:04 PM, Domenic Denicola  wrote:
> To be clear:
>
> Everyone can imagine use cases for playing videos backward. However, so far 
> the only statements we have about implementations are negative. My subthread 
> was more concerned with making the spec reflect current reality. If you can 
> convince implementers to support backward videos, then that's separate, and 
> we can change the spec again.
>

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> 
> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:
> 
>> QuickTime supports full variable speed playback and has done for well over
>> a decade. With bidirectionally predicted frames you need a fair few buffers
>> anyway, so generalising to full variable wait is easier than posters above
>> claim - you need to work a GOP at a time, but memory buffering isn't the
>> big issue these days.
>> 
> 
> "GOP”?

Group of Pictures.  Video-speak for the run between random access points.

> 
> How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
> keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> bytes = 4.32 GiB. Reading back those frames would kill performance so that
> all has to stay in VRAM. I respectfully deny that in such a case, memory
> buffering "isn't a big issue”.

well, 10s is a pretty long random access interval.

> 
> Now that I think about it, I guess there are more complicated strategies
> available that would reduce memory usage at the expense of repeated
> decoding.

which indeed QuickTime implemented around 10 years ago.

> E.g. in a first pass, decode forward and store every Nth frame.
> Then as you play backwards you need only redecode N-1 intermediate frames
> at time. I don't know whether HW decoder interfaces would actually let you
> implement that though...
> 
> What QuickTime got right was having a ToC approach to video so being able
>> to seek rapidly was possible without thrashing , whereas the stream
>> oriented approaches we are stuck with no wean knowing which bit of the file
>> to read to get the previous GOP is the hard part.
>> 
> 
> I don't understand. Can you explain this in more detail?

The movie file structure (and hence MP4) has a table-of-contents approach to 
file structure; each frame has its timestamps, file location, size, and 
keyframe-nature stored in compact tables in the head of the file.  This makes 
trick modes and so on easier; you’re not reading the actual video to seek for a 
keyframe, and so on.

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

On Tue, Sep 1, 2015 at 10:55 AM, David Singer  wrote:

>
> > On Sep 1, 2015, at 10:47 , Yay295  wrote:
> >
> > On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:
> > > On Sep 1, 2015, at 4:03 , Robert O'Callahan 
> wrote:
> > >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks 
> wrote:
> > >> QuickTime supports full variable speed playback and has done for well
> over
> > >> a decade. With bidirectionally predicted frames you need a fair few
> buffers
> > >> anyway, so generalising to full variable wait is easier than posters
> above
> > >> claim - you need to work a GOP at a time, but memory buffering isn't
> the
> > >> big issue these days.
> > >
> > > "GOP”?
> >
> > Group of Pictures.  Video-speak for the run between random access points.
> >
> > > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x
> 2
> > > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > > all has to stay in VRAM. I respectfully deny that in such a case,
> memory
> > > buffering "isn't a big issue”.
> >
> > well, 10s is a pretty long random access interval.
> >
> > There's no way to know the distance between keyframes though. The video
> could technically have only one keyframe and still work as a video.
>
> yes, but that is rare. There are indeed videos that don’t play well
> backward, or consume lots of memory and/or CPU, but most are fine.
>
> >
> > >> What QuickTime got right was having a ToC approach to video so being
> able
> > >> to seek rapidly was possible without thrashing , whereas the stream
> > >> oriented approaches we are stuck with no wean knowing which bit of
> the file
> > >> to read to get the previous GOP is the hard part.
> > >
> > > I don't understand. Can you explain this in more detail?
>

I explained the essential difference a while ago here:
http://lists.xiph.org/pipermail/vorbis-dev/2001-October/004846.html

The QuickTime file format defines movies that have tracks made of media;
the tracks are en edit list on the media; the media have the frame layout
information encoded.


> >
> > The movie file structure (and hence MP4) has a table-of-contents
> approach to file structure; each frame has its timestamps, file location,
> size, and keyframe-nature stored in compact tables in the head of the file.
> This makes trick modes and so on easier; you’re not reading the actual
> video to seek for a keyframe, and so on.
> >
> > I suppose the browser could generate this data the first time it reads
> through the video. It would use a lot less memory. Though that sounds like
> a problem for the browsers to solve, not the standard.
>
> There is no *generation* on the browser side; these tables are part of the
> file format.


Well, when it imports stream-oriented media it has to construct these in
memory, but they can be saved out again. I know that in theory this made
its way into the mp4 format, but I'm not sure how much of it is real.

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

On Tue, Sep 1, 2015 at 11:57 AM, David Singer  wrote:

>
> > On Sep 1, 2015, at 11:36 , Kevin Marks  wrote:
> > > I suppose the browser could generate this data the first time it reads
> through the video. It would use a lot less memory. Though that sounds like
> a problem for the browsers to solve, not the standard.
> >
> > There is no *generation* on the browser side; these tables are part of
> the file format.
> >
> > Well, when it imports stream-oriented media it has to construct these in
> memory, but they can be saved out again. I know that in theory this made
> its way into the mp4 format, but I'm not sure how much of it is real.
>
> Two different questions:
> a) do the QuickTime movie file format and the MP4 format contain these
> tables?  Yes.
> b) if I open another format, what happens?
>
> For case (a), the situation may be more nuanced if Movie Fragments are in
> use (you then get the tables for each fragment of the movie, though they
> are easily coalesced as they arrive).
>
> For case (b), classic QuickTime used to ‘convert to movie’ in memory,
> building the tables.  The situation is more nuanced on more recent engines.
>
> I think the point of the discussion is that one cannot dismiss trick modes
> such as reverse play as being unimplementable.


The other point for me is that given http://aomedia.org/ announcing plans
to create a new video file format to fix everything, that this time we
actually learn from this history and make one that is editable and seekable
again.

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 11:36 , Kevin Marks  wrote:
> > I suppose the browser could generate this data the first time it reads 
> > through the video. It would use a lot less memory. Though that sounds like 
> > a problem for the browsers to solve, not the standard.
> 
> There is no *generation* on the browser side; these tables are part of the 
> file format.
> 
> Well, when it imports stream-oriented media it has to construct these in 
> memory, but they can be saved out again. I know that in theory this made its 
> way into the mp4 format, but I'm not sure how much of it is real.

Two different questions:
a) do the QuickTime movie file format and the MP4 format contain these tables?  
Yes.
b) if I open another format, what happens?

For case (a), the situation may be more nuanced if Movie Fragments are in use 
(you then get the tables for each fragment of the movie, though they are easily 
coalesced as they arrive).

For case (b), classic QuickTime used to ‘convert to movie’ in memory, building 
the tables.  The situation is more nuanced on more recent engines.

I think the point of the discussion is that one cannot dismiss trick modes such 
as reverse play as being unimplementable. 

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:

> > On Sep 1, 2015, at 4:03 , Robert O'Callahan 
> wrote:
> >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks 
> wrote:
> >> QuickTime supports full variable speed playback and has done for well
> over
> >> a decade. With bidirectionally predicted frames you need a fair few
> buffers
> >> anyway, so generalising to full variable wait is easier than posters
> above
> >> claim - you need to work a GOP at a time, but memory buffering isn't the
> >> big issue these days.
> >
> > "GOP”?
>
> Group of Pictures.  Video-speak for the run between random access points.
>
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
>
> well, 10s is a pretty long random access interval.
>

There's no way to know the distance between keyframes though. The video
could technically have only one keyframe and still work as a video.


> >> What QuickTime got right was having a ToC approach to video so being
> able
> >> to seek rapidly was possible without thrashing , whereas the stream
> >> oriented approaches we are stuck with no wean knowing which bit of the
> file
> >> to read to get the previous GOP is the hard part.
> >
> > I don't understand. Can you explain this in more detail?
>
> The movie file structure (and hence MP4) has a table-of-contents approach
> to file structure; each frame has its timestamps, file location, size, and
> keyframe-nature stored in compact tables in the head of the file. This
> makes trick modes and so on easier; you’re not reading the actual video to
> seek for a keyframe, and so on.
>

I suppose the browser could generate this data the first time it reads
through the video. It would use a lot less memory. Though that sounds like
a problem for the browsers to solve, not the standard.

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 10:47 , Yay295  wrote:
> 
> On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:
> > On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:
> >> QuickTime supports full variable speed playback and has done for well over
> >> a decade. With bidirectionally predicted frames you need a fair few buffers
> >> anyway, so generalising to full variable wait is easier than posters above
> >> claim - you need to work a GOP at a time, but memory buffering isn't the
> >> big issue these days.
> >
> > "GOP”?
> 
> Group of Pictures.  Video-speak for the run between random access points.
> 
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
> 
> well, 10s is a pretty long random access interval.
> 
> There's no way to know the distance between keyframes though. The video could 
> technically have only one keyframe and still work as a video.

yes, but that is rare. There are indeed videos that don’t play well backward, 
or consume lots of memory and/or CPU, but most are fine.

>  
> >> What QuickTime got right was having a ToC approach to video so being able
> >> to seek rapidly was possible without thrashing , whereas the stream
> >> oriented approaches we are stuck with no wean knowing which bit of the file
> >> to read to get the previous GOP is the hard part.
> >
> > I don't understand. Can you explain this in more detail?
> 
> The movie file structure (and hence MP4) has a table-of-contents approach to 
> file structure; each frame has its timestamps, file location, size, and 
> keyframe-nature stored in compact tables in the head of the file. This makes 
> trick modes and so on easier; you’re not reading the actual video to seek for 
> a keyframe, and so on.
> 
> I suppose the browser could generate this data the first time it reads 
> through the video. It would use a lot less memory. Though that sounds like a 
> problem for the browsers to solve, not the standard.

There is no *generation* on the browser side; these tables are part of the file 
format.

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Wed, Sep 2, 2015 at 5:30 AM, David Singer  wrote:

> On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
>
> well, 10s is a pretty long random access interval.
>

It's easy to find sources on the Internet advising people to use 10s
keyframe intervals.

> Now that I think about it, I guess there are more complicated strategies
> > available that would reduce memory usage at the expense of repeated
> > decoding.
>
> which indeed QuickTime implemented around 10 years ago.
>

It appears that most platform and HW decoder interfaces are incompatible
with this strategy, so in practice implementing this across platforms is
still a big problem.

Nevertheless we can hope for that situation to improve, and negative
playback rates are implementable for some videos, so it makes sense to me
to leave negative playback rates in the spec.

The movie file structure (and hence MP4) has a table-of-contents approach
> to file structure; each frame has its timestamps, file location, size, and
> keyframe-nature stored in compact tables in the head of the file.  This
> makes trick modes and so on easier; you’re not reading the actual video to
> seek for a keyframe, and so on.
>

I think every important video container format has some kind of keyframe
directory.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

QuickTime supports full variable speed playback and has done for well over
a decade. With bidirectionally predicted frames you need a fair few buffers
anyway, so generalising to full variable wait is easier than posters above
claim - you need to work a GOP at a time, but memory buffering isn't the
big issue these days.
What QuickTime got right was having a ToC approach to video so being able
to seek rapidly was possible without thrashing , whereas the stream
oriented approaches we are stuck with no wean knowing which bit of the file
to read to get the previous GOP is the hard part.

On Fri, Aug 28, 2015 at 6:02 PM, Xidorn Quan  wrote:

> On Sat, Aug 29, 2015 at 8:27 AM, Robert O'Callahan 
> wrote:
> > On Sat, Aug 29, 2015 at 8:18 AM, James Ross 
> wrote:
> >
> >> Support is certainly poor; Internet Explorer/Trident and Edge both
> support
> >> negative playback rates on desktop (I haven’t tested mobile) but do so
> by
> >> simply showing the key frames as they are reached in reverse, in my
> testing.
> >
> > That's not so hard to implement, but it's also mostly useless since
> > keyframes are often several seconds apart or more.
>
> It could be useful for a few usecases like fast-backward. Windows
> Media Player does it this way.
>
> FWIW, QuickTime supports per-frame backward playback if you press and
> hold the left arrow. I guess they cannot guarantee the rate, which
> makes them require holding the key instead of providing a playback
> rate setting.
>
> - Xidorn
>

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>>
>> ...
>>
>>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
>> content.
>
> That's really helpful, thanks. Combined with Edge's keyframes-only support, 
> it sounds like we should probably leave the spec as it is.
>
> Do you have thoughts on a mozPreservesPitch equivalent?

Should we just standardize HTMLMediaElement.preservesPitch, perhaps?
Note that WebKit also has webkitPreservesPitch, but I removed it from
Blink because it didn't actually do anything in Chromium.

In both Gecko and WebKit it defaults to true. Is there anything else
worth knowing before writing the spec for this?

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:

> QuickTime supports full variable speed playback and has done for well over
> a decade. With bidirectionally predicted frames you need a fair few buffers
> anyway, so generalising to full variable wait is easier than posters above
> claim - you need to work a GOP at a time, but memory buffering isn't the
> big issue these days.
>

"GOP"?

How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
bytes = 4.32 GiB. Reading back those frames would kill performance so that
all has to stay in VRAM. I respectfully deny that in such a case, memory
buffering "isn't a big issue".

Now that I think about it, I guess there are more complicated strategies
available that would reduce memory usage at the expense of repeated
decoding. E.g. in a first pass, decode forward and store every Nth frame.
Then as you play backwards you need only redecode N-1 intermediate frames
at time. I don't know whether HW decoder interfaces would actually let you
implement that though...

What QuickTime got right was having a ToC approach to video so being able
> to seek rapidly was possible without thrashing , whereas the stream
> oriented approaches we are stuck with no wean knowing which bit of the file
> to read to get the previous GOP is the hard part.
>

I don't understand. Can you explain this in more detail?

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Michael Enright]

On Fri, Aug 28, 2015 at 11:17 AM, Domenic Denicola  wrote:
> From:  Robert O'Callahan
>
>> According to the spec it should work, but it's very low priority for us and
>> implementing it would be very inefficient as Yay295 describes. So I don't
>> think it's going to happen in Firefox in the forseeable future.
>
> I was looking in to this yesterday and it seems like the spec places absolute 
> no limits on playbackRate. Am I right? This seems a bit bad.
>
> If nobody is supporting negative nor has any plans to, we should at least 
> consider throwing for negative. I guess we can leave the upper limit 
> unspecified, unless implementations all happen to agree on one.
>

I have implemented video element implementations which allowed the
video to play in reverse, and scripts that ran on such implementations
that allowed the user to ask for playback in reverse. If it is thought
that browser scripts need guidance on what speeds are possible for a
given server or video, then by all means go that route.

USE CASE: Implementing things like VidiPath or in some other fashion
implementing satellte STB or cable STB UI in HTML 5.

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/31/15, Michael Enright  wrote:
> On Fri, Aug 28, 2015 at 11:17 AM, Domenic Denicola  wrote:
>> From:  Robert O'Callahan
>>
>>> According to the spec it should work, but it's very low priority for us
>>> and
>>> implementing it would be very inefficient as Yay295 describes. So I
>>> don't
>>> think it's going to happen in Firefox in the forseeable future.
>>
>> I was looking in to this yesterday and it seems like the spec places
>> absolute no limits on playbackRate. Am I right? This seems a bit bad.
>>
>> If nobody is supporting negative nor has any plans to, we should at least
>> consider throwing for negative. I guess we can leave the upper limit
>> unspecified, unless implementations all happen to agree on one.
>>
>
> I have implemented video element implementations which allowed the
> video to play in reverse, and scripts that ran on such implementations
> that allowed the user to ask for playback in reverse. If it is thought
> that browser scripts need guidance on what speeds are possible for a
> given server or video, then by all means go that route.
>
> USE CASE: Implementing things like VidiPath or in some other fashion
> implementing satellte STB or cable STB UI in HTML 5.
>

USE CASE: https://www.youtube.com/watch?v=BtSdt1G6mcE
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

To be clear:

Everyone can imagine use cases for playing videos backward. However, so far the 
only statements we have about implementations are negative. My subthread was 
more concerned with making the spec reflect current reality. If you can 
convince implementers to support backward videos, then that's separate, and we 
can change the spec again.

Re: [whatwg] VIDEO and pitchAdjustment

[Eric Carlson]

> On Aug 31, 2015, at 12:04 PM, Domenic Denicola  wrote:
> 
> My subthread was more concerned with making the spec reflect current reality. 
> If you can convince implementers to support backward videos, then that's 
> separate, and we can change the spec again.
> 


  FWIW, Safari supports negative playback rates on the desktop and on iOS. 


> On Aug 27, 2015, at 11:02 AM, Garrett Smith  wrote:
> 
> Negative playbackRate, to watch videos backwards, currently crashes
> Safari 8


  The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE 
content.

eric

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: Eric Carlson [mailto:eric.carl...@apple.com]

>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>
> ...
>
>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
> content.

That's really helpful, thanks. Combined with Edge's keyframes-only support, it 
sounds like we should probably leave the spec as it is.

Do you have thoughts on a mozPreservesPitch equivalent?

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Sat, Aug 29, 2015 at 8:18 AM, James Ross ja...@james-ross.co.uk wrote:

 Support is certainly poor; Internet Explorer/Trident and Edge both support
 negative playback rates on desktop (I haven’t tested mobile) but do so by
 simply showing the key frames as they are reached in reverse, in my testing.


That's not so hard to implement, but it's also mostly useless since
keyframes are often several seconds apart or more.


 Firefox, Chrome and Safari on desktop and mobile don’t support negative
 values at all AFAICT. I have notes here suggesting that mobile platforms
 don’t even support positive rates other that 1.


I'm pretty sure mobile Firefox supports non-1 playback rates.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Robert 
O'Callahan

 According to the spec it should work, but it's very low priority for us and
 implementing it would be very inefficient as Yay295 describes. So I don't
 think it's going to happen in Firefox in the forseeable future.

I was looking in to this yesterday and it seems like the spec places absolute 
no limits on playbackRate. Am I right? This seems a bit bad.

If nobody is supporting negative nor has any plans to, we should at least 
consider throwing for negative. I guess we can leave the upper limit 
unspecified, unless implementations all happen to agree on one.

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Fri, Aug 28, 2015 at 12:17 PM, Domenic Denicola d...@domenic.me wrote:

 From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Robert
 O'Callahan
  According to the spec it should work, but it's very low priority for us
 and
  implementing it would be very inefficient as Yay295 describes. So I don't
  think it's going to happen in Firefox in the forseeable future.

 I was looking in to this yesterday and it seems like the spec places
 absolute no limits on playbackRate. Am I right? This seems a bit bad.

 If nobody is supporting negative nor has any plans to, we should at least
 consider throwing for negative. I guess we can leave the upper limit
 unspecified, unless implementations all happen to agree on one.


playbackRate should also probably be set to 0 when the video is paused,
seeing as they accomplish the same task. Or better yet redefine pause() to
be setting playbackRate to 0. You would of course then have to keep a note
of what the playbackRate was before pause() was called, but it would fix
the issue of playbackRate being set to zero, yet video.paused returning
false. So video.pause() would save the current playbackRate and then set
playbackRate to 0, and video.play() would restore the playbackRate saved by
pause(). video.paused could then just return (playbackRate ? false : true).

Re: [whatwg] VIDEO and pitchAdjustment

[Xidorn Quan]

On Sat, Aug 29, 2015 at 8:27 AM, Robert O'Callahan rob...@ocallahan.org wrote:
 On Sat, Aug 29, 2015 at 8:18 AM, James Ross ja...@james-ross.co.uk wrote:

 Support is certainly poor; Internet Explorer/Trident and Edge both support
 negative playback rates on desktop (I haven’t tested mobile) but do so by
 simply showing the key frames as they are reached in reverse, in my testing.

 That's not so hard to implement, but it's also mostly useless since
 keyframes are often several seconds apart or more.

It could be useful for a few usecases like fast-backward. Windows
Media Player does it this way.

FWIW, QuickTime supports per-frame backward playback if you press and
hold the left arrow. I guess they cannot guarantee the rate, which
makes them require holding the key instead of providing a playback
rate setting.

- Xidorn

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Thu, Aug 27, 2015 at 12:02 PM, Garrett Smith dhtmlkitc...@gmail.com
wrote:

 Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


It would be much easier to re-encode the videos in reverse and provide two
versions of the video. Videos are made to be played in one direction, and
they use this assumption to improve their compression efficiency.
Basically, videos have two different types of frames; let's call them Full
frames and Partial frames. A Full frame contains all of the information
that a stand-alone image would have. If a video had only a single Full
frame, it would be able to display that image and show it for the length of
the video (incidentally, this is what a WebP image is). Partial frames, on
the other hand, do NOT have enough information to produce a full image.
They merely contain the information representing the difference between the
previous frame and the current frame. This obviously saves a lot of space
if the frames are similar. A video will contain a mixture of Full frames
and Partial frames, with many more Partial frames than Full frames (ex. 1
Full frame followed by 10 seconds of Partial frames (~240 partial frames)).
What this means is that when you play a video in the proper direction, it
must first decode a Full frame, but then it only has to decode Partial
frames (until the next Full frame). However, if you try to play a video in
reverse, it must decode a Full frame and ALL of the Partial frames up to
the current frame before it can display the current frame. To display the
frame before that, it must decode all of those frames again, and so on for
every frame of the video. You could of course store the decoded frames in
memory, but that will take a lot of space. 240 uncompressed frames at 720p
is over half a gigabyte of information. It's just not practical.

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith dhtmlkitc...@gmail.com
wrote:

 It would be useful to have pitch adjustment for VIDEO element. There
 is playbackRate, to control playback speed — useful!* And there is
 vv.mozPreservesPitch, in Firefox, which can be set to false, so that
 pitch will adjust to the speed of the video, sort of like old analog
 gear (tapes and records).


We should standardize mozPreservesPitch. I'm embarrassed that we haven't
done so already.

Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


According to the spec it should work, but it's very low priority for us and
implementing it would be very inefficient as Yay295 describes. So I don't
think it's going to happen in Firefox in the forseeable future.

But variable pitch control it would be useful for music adjustments
 like over the mountain,  Black Star, Take your Whiskey Home, all
 originally recorded at half-step detuning (A=415), none of which match
 standard tuning commonly used in schools, industry, etc. Recordings
 of Baroque era music often use instruments tuned lower (and also
 different scale temperment, but that is a different issue). So it
 would be nice to adjust the pitch with a `pitchAdjustment` property,
 as a double, to adjust pitch in cents.


I'd prefer to focus on making sure that Web Audio's Audio Workers are
powerful enough for Web developers to implement this themselves.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/27/15, Robert O'Callahan rob...@ocallahan.org wrote:
 On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith dhtmlkitc...@gmail.com
 wrote:

 It would be useful to have pitch adjustment for VIDEO element. There
 is playbackRate, to control playback speed — useful!* And there is
 vv.mozPreservesPitch, in Firefox, which can be set to false, so that
 pitch will adjust to the speed of the video, sort of like old analog
 gear (tapes and records).


 We should standardize mozPreservesPitch. I'm embarrassed that we haven't
 done so already.

 Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


 According to the spec it should work, but it's very low priority for us and
 implementing it would be very inefficient as Yay295 describes. So I don't
 think it's going to happen in Firefox in the forseeable future.

 But variable pitch control it would be useful for music adjustments
 like over the mountain,  Black Star, Take your Whiskey Home, all
 originally recorded at half-step detuning (A=415), none of which match
 standard tuning commonly used in schools, industry, etc. Recordings
 of Baroque era music often use instruments tuned lower (and also
 different scale temperment, but that is a different issue). So it
 would be nice to adjust the pitch with a `pitchAdjustment` property,
 as a double, to adjust pitch in cents.


 I'd prefer to focus on making sure that Web Audio's Audio Workers are
 powerful enough for Web developers to implement this themselves.


I don't know enough about Web Audio to comment on that. Most of what
I'm using is on YouTube.  And there are some video issues that I'd
like to comment on because it's such a common situation to have audio
as video MP4's. So here's two video user scenarios:

Want hear the audio at the desired pitch, while simultaneously
watching the video at the desired speed, to learn it.

For fine-grained adjustment for music performances that are in between
keys (off by a few cents) or need to be transcribed.

This guy's cover of Over the Mountain is pretty good.
https://www.youtube.com/watch?v=Ze6h1t7Z734 Right now, I can get that
up a half step to standard tuning by increasing the speed and setting
mozPreservesPitch:—

var vv = document.querySelector(video);
vv.mozPreservesPitch = false;
vv.playbackRate = 1.05;

But it's a little sped up. Not ideal for learning! I'd rather slow
down it a bit. Enter the Amazing Slow Downer … does what I want.

OK, there's my two user scenarios with an example. Thanks for reading!
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] video feedback

[Ian Hickson]

On Thu, 20 Dec 2012, Jer Noble wrote:
 On Dec 17, 2012, at 4:01 PM, Ian Hickson i...@hixie.ch wrote:
  
  Should we add a preciseSeek() method with two arguments that does a 
  seek using the given rational time?
 
 This method would be more useful if there were a way to retrieve the 
 media's time scale.  Otherwise, the script would have to pick an 
 arbitrary scale value, or provide the correct media scale through other 
 means (such as querying the server hosting the media).  Additionally, 
 authors like Rob are going to want to retrieve this precise 
 representation of the currentTime.  If rational time values were 
 encapsulated into their own interface, a preciseCurrentTime (or 
 similar) read-write attribute could be used instead.

Ok. I assume this is something you (Apple) are interested in implementing; 
is this something any other browser vendors want to support? If so, I'll 
be happy to add something along these lines.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video feedback

[Mark Callow]

On 2012/12/18 9:01, Ian Hickson wrote:
 On Tue, 2 Oct 2012, Jer Noble wrote:
 The nature of floating point math makes precise frame navigation 
 difficult, if not impossible.  Rob's test is especially hairy, given 
 that each frame has a timing bound of [startTime, endTime), and his test 
 attempts to navigate directly to the startTime of a given frame, a value 
 which gives approximately zero room for error.

 ...
 That makes sense.

 Should we add a preciseSeek() method with two arguments that does a seek 
 using the given rational time?


I draw your attention to Don't Store that in a float
http://randomascii.wordpress.com/2012/02/13/dont-store-that-in-a-float/
and its suggestion to use a double starting at 2^32 to avoid the issue
around precision changing with magnitude as the time increases.

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] video feedback

[Ian Hickson]

On Thu, 20 Dec 2012, Mark Callow wrote:
 On 2012/12/18 9:01, Ian Hickson wrote:
  On Tue, 2 Oct 2012, Jer Noble wrote:
  The nature of floating point math makes precise frame navigation 
  difficult, if not impossible.  Rob's test is especially hairy, given 
  that each frame has a timing bound of [startTime, endTime), and his 
  test attempts to navigate directly to the startTime of a given frame, 
  a value which gives approximately zero room for error.
 
  That makes sense.
 
  Should we add a preciseSeek() method with two arguments that does a 
  seek using the given rational time?
 
 I draw your attention to Don't Store that in a float 
 http://randomascii.wordpress.com/2012/02/13/dont-store-that-in-a-float/ 
 and its suggestion to use a double starting at 2^32 to avoid the issue 
 around precision changing with magnitude as the time increases.

Everything in the Web platform already uses doubles.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video feedback

[Boris Zbarsky]

On 12/20/12 9:54 AM, Ian Hickson wrote:

Everything in the Web platform already uses doubles.


Except WebGL.  And Audio API wave tables, sample rates, AudioParams, PCM 
data (though thankfully times in Audio API do use doubles).  And 
graphics libraries used to implement canvas, in many cases...


I think the only safe claim about everything in the web platform is 
that it's all different.  ;)


-Boris

Re: [whatwg] video feedback

[Mark Callow]

On 2012/12/21 2:54, Ian Hickson wrote:
 On Thu, 20 Dec 2012, Mark Callow wrote:
 I draw your attention to Don't Store that in a float 
 http://randomascii.wordpress.com/2012/02/13/dont-store-that-in-a-float/ 
 and its suggestion to use a double starting at 2^32 to avoid the issue 
 around precision changing with magnitude as the time increases.
 Everything in the Web platform already uses doubles.
Yes, except as noted by Boris. The important point is the idea of using
2^32 as zero time which means the precision barely changes across the
range of time values of interest to games, videos, etc.

Regards

-Mark

-- 
注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合
が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情
報の使用を固く禁じております。エラー、手違いでこのメールを受け取られまし
たら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and
privileged information from HI Corporation. If you are not the intended
recipient, any disclosure, photocopying, distribution or use of the
contents of the received information is prohibited. If you have received
this e-mail in error, please notify the sender immediately and
permanently delete this message and all related copies.

Re: [whatwg] video feedback

[Ian Hickson]

On Fri, 21 Dec 2012, Mark Callow wrote:
 On 2012/12/21 2:54, Ian Hickson wrote:
  On Thu, 20 Dec 2012, Mark Callow wrote:
  I draw your attention to Don't Store that in a float 
  http://randomascii.wordpress.com/2012/02/13/dont-store-that-in-a-float/ 
  and its suggestion to use a double starting at 2^32 to avoid the issue 
  around precision changing with magnitude as the time increases.
  Everything in the Web platform already uses doubles.
 Yes, except as noted by Boris. The important point is the idea of using 
 2^32 as zero time which means the precision barely changes across the 
 range of time values of interest to games, videos, etc.

Ah, well, for video that ship has sailed, really.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video feedback

[Jer Noble]

On Dec 20, 2012, at 7:27 PM, Mark Callow callow.m...@artspark.co.jp wrote:

 On 2012/12/21 2:54, Ian Hickson wrote:
 On Thu, 20 Dec 2012, Mark Callow wrote:
 I draw your attention to Don't Store that in a float 
 http://randomascii.wordpress.com/2012/02/13/dont-store-that-in-a-float/ 
 and its suggestion to use a double starting at 2^32 to avoid the issue 
 around precision changing with magnitude as the time increases.
 Everything in the Web platform already uses doubles.
 Yes, except as noted by Boris. The important point is the idea of using 2^32 
 as zero time which means the precision barely changes across the range of 
 time values of interest to games, videos, etc. 

I don't believe the frame accuracy problem in question had to do with 
precision instability, per se.  Many of Rob Coenen's frame accuracy issues were 
found within the first second of video.  Admittedly, this is where the 
avaliable precision is changing most rapidly, but it is also where available 
precision is greatest by far.

An integral rational number has a benefit over even the 2^32 zero time 
suggestion: for common time scale values[1], it is intrinsically stable over 
the range of time t=[0..2^43).  It has the added benefit of being exactly the 
representation used by the underlying media engine.


On Dec 17, 2012, at 4:01 PM, Ian Hickson i...@hixie.ch wrote:

 Should we add a preciseSeek() method with two arguments that does a seek 
 using the given rational time?


This method would be more useful if there were a way to retrieve the media's 
time scale.  Otherwise, the script would have to pick an arbitrary scale value, 
or provide the correct media scale through other means (such as querying the 
server hosting the media).  Additionally, authors like Rob are going to want to 
retrieve this precise representation of the currentTime.  If rational time 
values were encapsulated into their own interface, a preciseCurrentTime (or 
similar) read-write attribute could be used instead.

-Jer

[i] E.g., 1001 is a common time scale for 29.997 and 23.976 FPS video.

Re: [whatwg] video feedback

[Ian Hickson]

On Tue, 2 Oct 2012, Jer Noble wrote:
 On Sep 17, 2012, at 12:43 PM, Ian Hickson i...@hixie.ch wrote:
  On Mon, 9 Jul 2012, adam k wrote:
 
  i'm aware that crooked framerates (i.e. the notorious 29.97) were not 
  supported when frame accuracy was implemented.  in my tests, 29.97DF 
  timecodes were incorrect by 1 to 3 frames at any given point.
  
  will there ever be support for crooked framerate accuracy?  i would 
  be more than happy to contribute whatever i can to help test it and 
  make it possible.  can someone comment on this?
  
  This is a Quality of Implementation issue, basically. I believe 
  there's nothing inherently in the API that would make accuracy to such 
  timecodes impossible.
 
 The nature of floating point math makes precise frame navigation 
 difficult, if not impossible.  Rob's test is especially hairy, given 
 that each frame has a timing bound of [startTime, endTime), and his test 
 attempts to navigate directly to the startTime of a given frame, a value 
 which gives approximately zero room for error.
 
 I'm most familiar with MPEG containers, but I believe the following is 
 also true of the WebM container: times are represented by a rational 
 number, timeValue / timeScale, where both numerator and denominator are 
 unsigned integers.  To seek to a particular media time, we must convert 
 a floating-point time value into this rational time format (e.g. when 
 calculating the 4th frame's start time, from 3 * 1/29.97 to 3 * 
 1001/3).  If there is a floating-point error in the wrong direction 
 (e.g., as above, a numerator of 3002 vs 3003), the end result will not 
 be the frame's startTime, but one timeScale before it.
 
 We've fixed some frame accuracy bugs in WebKit (and Chromium) by 
 carefully rounding the incoming floating point time value, taking into 
 account the media's time scale, and rounding to the nearest 1/timeScale 
 value.  This fixes Rob's precision test, but at the expense of 
 precision. (I.e. in a 30 fps movie, currentTime = 0.99 / 30 will 
 navigate to the second frame, not the first, due to rounding, which is 
 technically incorrect.)
 
 This is a common problem, and Apple media frameworks (for example) 
 therefore provide rational time classes which provide enough accuracy 
 for precise navigation (e.g. QTTime, CMTime). Using a floating point 
 number to represent time with any precision is not generally accepted as 
 good practice when these rational time classes are available.

That makes sense.

Should we add a preciseSeek() method with two arguments that does a seek 
using the given rational time?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video feedback

[Jer Noble]

On Sep 17, 2012, at 12:43 PM, Ian Hickson i...@hixie.ch wrote:

 On Mon, 9 Jul 2012, adam k wrote:
 
 i have a 25fps video, h264, with a burned in timecode.  it seems to be 
 off by 1 frame when i compare the burned in timecode to the calculated 
 timecode.  i'm using rob coenen's test app at 
 http://www.massive-interactive.nl/html5_video/smpte_test_universal.html 
 to load my own video.
 
 what's the process here to report issues?  please let me know whatever 
 formal or informal steps are required and i'll gladly follow them.
 
 Depends on the browser. Which browser?
 
 
 i'm aware that crooked framerates (i.e. the notorious 29.97) were not 
 supported when frame accuracy was implemented.  in my tests, 29.97DF 
 timecodes were incorrect by 1 to 3 frames at any given point.
 
 will there ever be support for crooked framerate accuracy?  i would be 
 more than happy to contribute whatever i can to help test it and make it 
 possible.  can someone comment on this?
 
 This is a Quality of Implementation issue, basically. I believe there's 
 nothing inherently in the API that would make accuracy to such timecodes 
 impossible.

TLDR; for precise navigation, you need to use a a rational time class, rather 
than a float value.

The nature of floating point math makes precise frame navigation difficult, if 
not impossible.  Rob's test is especially hairy, given that each frame has a 
timing bound of [startTime, endTime), and his test attempts to navigate 
directly to the startTime of a given frame, a value which gives approximately 
zero room for error.

I'm most familiar with MPEG containers, but I believe the following is also 
true of the WebM container: times are represented by a rational number, 
timeValue / timeScale, where both numerator and denominator are unsigned 
integers.  To seek to a particular media time, we must convert a floating-point 
time value into this rational time format (e.g. when calculating the 4th 
frame's start time, from 3 * 1/29.97 to 3 * 1001/3).  If there is a 
floating-point error in the wrong direction (e.g., as above, a numerator of 
3002 vs 3003), the end result will not be the frame's startTime, but one 
timeScale before it. 

We've fixed some frame accuracy bugs in WebKit (and Chromium) by carefully 
rounding the incoming floating point time value, taking into account the 
media's time scale, and rounding to the nearest 1/timeScale value.  This fixes 
Rob's precision test, but at the expense of precision. (I.e. in a 30 fps movie, 
currentTime = 0.99 / 30 will navigate to the second frame, not the first, 
due to rounding, which is technically incorrect.)

This is a common problem, and Apple media frameworks (for example) therefore 
provide rational time classes which provide enough accuracy for precise 
navigation (e.g. QTTime, CMTime). Using a floating point number to represent 
time with any precision is not generally accepted as good practice when these 
rational time classes are available.

-Jer

Re: [whatwg] video feedback

[Silvia Pfeiffer]

On Wed, Oct 3, 2012 at 6:41 AM, Jer Noble jer.no...@apple.com wrote:
 On Sep 17, 2012, at 12:43 PM, Ian Hickson i...@hixie.ch wrote:

 On Mon, 9 Jul 2012, adam k wrote:

 i have a 25fps video, h264, with a burned in timecode.  it seems to be
 off by 1 frame when i compare the burned in timecode to the calculated
 timecode.  i'm using rob coenen's test app at
 http://www.massive-interactive.nl/html5_video/smpte_test_universal.html
 to load my own video.

 what's the process here to report issues?  please let me know whatever
 formal or informal steps are required and i'll gladly follow them.

 Depends on the browser. Which browser?


 i'm aware that crooked framerates (i.e. the notorious 29.97) were not
 supported when frame accuracy was implemented.  in my tests, 29.97DF
 timecodes were incorrect by 1 to 3 frames at any given point.

 will there ever be support for crooked framerate accuracy?  i would be
 more than happy to contribute whatever i can to help test it and make it
 possible.  can someone comment on this?

 This is a Quality of Implementation issue, basically. I believe there's
 nothing inherently in the API that would make accuracy to such timecodes
 impossible.

 TLDR; for precise navigation, you need to use a a rational time class, rather 
 than a float value.

 The nature of floating point math makes precise frame navigation difficult, 
 if not impossible.  Rob's test is especially hairy, given that each frame has 
 a timing bound of [startTime, endTime), and his test attempts to navigate 
 directly to the startTime of a given frame, a value which gives approximately 
 zero room for error.

 I'm most familiar with MPEG containers, but I believe the following is also 
 true of the WebM container: times are represented by a rational number, 
 timeValue / timeScale, where both numerator and denominator are unsigned 
 integers.


FYI: the Ogg container also uses rational numbers to represent time.


  To seek to a particular media time, we must convert a floating-point time 
 value into this rational time format (e.g. when calculating the 4th frame's 
 start time, from 3 * 1/29.97 to 3 * 1001/3).  If there is a 
 floating-point error in the wrong direction (e.g., as above, a numerator of 
 3002 vs 3003), the end result will not be the frame's startTime, but one 
 timeScale before it.

 We've fixed some frame accuracy bugs in WebKit (and Chromium) by carefully 
 rounding the incoming floating point time value, taking into account the 
 media's time scale, and rounding to the nearest 1/timeScale value.  This 
 fixes Rob's precision test, but at the expense of precision. (I.e. in a 30 
 fps movie, currentTime = 0.99 / 30 will navigate to the second frame, 
 not the first, due to rounding, which is technically incorrect.)

 This is a common problem, and Apple media frameworks (for example) therefore 
 provide rational time classes which provide enough accuracy for precise 
 navigation (e.g. QTTime, CMTime). Using a floating point number to represent 
 time with any precision is not generally accepted as good practice when these 
 rational time classes are available.

 -Jer

Re: [whatwg] video preload implementation feedback

[Simon Pieters]

On Fri, 15 Jun 2012 18:01:09 +0200, Ian Hickson i...@hixie.ch wrote:


When preload=none, step 2 of
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#concept-media-load-resource
should not be optional.

The effective (internal) preload state should be defined.

It should also be defined that with preload=metadata, readyState should
never go beyond HAVE_CURRENT_DATA, even for a data: URL or otherwise
fully cached resource.


Please specify the above.

Making it non-conforming for a user agent to aggressively cache  
resources,

especially if the user has asked for it, is a non-starter.


Aggressively caching does not necessarily need to be exposed to scripts.


There are going
to be cases where that's what the user wants, and I don't see why we  
would

have to make this non-conforming.



 As far as I can tell, the spec is as detailed as it can be here given
 the range of possible implementation strategies that we need to allow.

 Could you give a concrete example of what you are concerned about?

video src=x preload=none onsuspend=makeSiteWork()/video


Then we should stop firing suspend in the preload=none case,


No.


or fire it
in every case if preload=non, even if the UA immediately unsuspends.


Maybe.


But I'm not convinced anyone is going to hook into onsuspend in this way.
There'd be no point as far as I can tell, and it's more complicated than
the alternative (just run the code straight away without waiting).


You're not convinced that people on the Web do things in ways more  
complicated and fragile than necessary, just because it's possible?


--
Simon Pieters
Opera Software

Re: [whatwg] video preload implementation feedback

[Simon Pieters]

On Thu, 14 Jun 2012 20:32:16 +0200, Ian Hickson i...@hixie.ch wrote:


On Thu, 14 Jun 2012, Simon Pieters wrote:


It's not more. But it still is. Even though images aren't required to
load at all, you still recently changed the way they load to be
compatible (http://html5.org/r/7128 ). We should also specify how videos
load to be compatible. We can do it now and get everyone to align on a
good behavior, or we can wait and do it in a few years when Web content
relies on what the market leader does, whether that's good or bad
behavior.


I don't understand what behaviour it is that you think we should define.


When preload=none, step 2 of  
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#concept-media-load-resource  
should not be optional.


The effective (internal) preload state should be defined.

It should also be defined that with preload=metadata, readyState should  
never go beyond HAVE_CURRENT_DATA, even for a data: URL or otherwise fully  
cached resource.



As far as I can tell, the spec is as detailed as it can be here given the
range of possible implementation strategies that we need to allow.

Could you give a concrete example of what you are concerned about?


video src=x preload=none onsuspend=makeSiteWork()/video

--
Simon Pieters
Opera Software

Re: [whatwg] video preload implementation feedback

[Ian Hickson]

On Fri, 15 Jun 2012, Simon Pieters wrote:
 On Thu, 14 Jun 2012 20:32:16 +0200, Ian Hickson i...@hixie.ch wrote:
  On Thu, 14 Jun 2012, Simon Pieters wrote:
   
   It's not more. But it still is. Even though images aren't required 
   to load at all, you still recently changed the way they load to be 
   compatible (http://html5.org/r/7128 ). We should also specify how 
   videos load to be compatible. We can do it now and get everyone to 
   align on a good behavior, or we can wait and do it in a few years 
   when Web content relies on what the market leader does, whether 
   that's good or bad behavior.
  
  I don't understand what behaviour it is that you think we should 
  define.
 
 When preload=none, step 2 of 
 http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#concept-media-load-resource
  
 should not be optional.
 
 The effective (internal) preload state should be defined.
 
 It should also be defined that with preload=metadata, readyState should 
 never go beyond HAVE_CURRENT_DATA, even for a data: URL or otherwise 
 fully cached resource.

Making it non-conforming for a user agent to aggressively cache resources, 
especially if the user has asked for it, is a non-starter. There are going 
to be cases where that's what the user wants, and I don't see why we would 
have to make this non-conforming.


  As far as I can tell, the spec is as detailed as it can be here given 
  the range of possible implementation strategies that we need to allow.
  
  Could you give a concrete example of what you are concerned about?
 
 video src=x preload=none onsuspend=makeSiteWork()/video

Then we should stop firing suspend in the preload=none case, or fire it 
in every case if preload=non, even if the UA immediately unsuspends.

But I'm not convinced anyone is going to hook into onsuspend in this way. 
There'd be no point as far as I can tell, and it's more complicated than 
the alternative (just run the code straight away without waiting).

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video preload implementation feedback

[Simon Pieters]

On Thu, 14 Jun 2012 00:57:29 +0200, Ian Hickson i...@hixie.ch wrote:


On Wed, 9 May 2012, Simon Pieters wrote:

On Tue, 08 May 2012 18:59:29 +0200, Ian Hickson i...@hixie.ch wrote:
 On Thu, 18 Aug 2011, Philip Jägenstedt wrote:
 
  This is true, but as long as a few big browsers implement e.g.
  preload=none in a somewhat compatible way, it's hard to imagine
  page authors not coming to depend on that behavior so that it
  becomes required for web compat. It would be interesting to know if
  there are counter-examples, any script-visible behavior that is
  allowed to vary greatly between implementations without causing
  scripts to break.

 Images aren't required to load at all. Scripts aren't required to run
 at all. The window size is allowed to be any dimension at all. CSS
 isn't required to be supported at all. Users are allowed to apply
 arbitrary user style sheets. Users are allowed to interact with form
 controls by using the keyboard or the mouse or any other input device.

 All of these do break some pages.

That CSS is optional and that users are allowed to apply user style
sheets didn't stop you from specifying the Rendering section in great
detail.


Optional detail. UAs aren't required to follow that section.



Making video behavior underdefined just because users should be able
to disable video loading in preferences just means that in a few years
the behavior of the market leader needs to be reverse engineered and
implemented by everyone else.


I do not understand how this particular feature could end up in that
state any more than the other features I list above.


It's not more. But it still is. Even though images aren't required to load  
at all, you still recently changed the way they load to be compatible  
(http://html5.org/r/7128 ). We should also specify how videos load to be  
compatible. We can do it now and get everyone to align on a good behavior,  
or we can wait and do it in a few years when Web content relies on what  
the market leader does, whether that's good or bad behavior.


--
Simon Pieters
Opera Software

Re: [whatwg] video preload implementation feedback

[Ian Hickson]

On Thu, 14 Jun 2012, Simon Pieters wrote:
 
 It's not more. But it still is. Even though images aren't required to 
 load at all, you still recently changed the way they load to be 
 compatible (http://html5.org/r/7128 ). We should also specify how videos 
 load to be compatible. We can do it now and get everyone to align on a 
 good behavior, or we can wait and do it in a few years when Web content 
 relies on what the market leader does, whether that's good or bad 
 behavior.

I don't understand what behaviour it is that you think we should define. 
As far as I can tell, the spec is as detailed as it can be here given the 
range of possible implementation strategies that we need to allow.

Could you give a concrete example of what you are concerned about?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video preload implementation feedback

[Ian Hickson]

On Wed, 9 May 2012, Simon Pieters wrote:
 On Tue, 08 May 2012 18:59:29 +0200, Ian Hickson i...@hixie.ch wrote:
  On Thu, 18 Aug 2011, Philip Jägenstedt wrote:
   
   This is true, but as long as a few big browsers implement e.g. 
   preload=none in a somewhat compatible way, it's hard to imagine 
   page authors not coming to depend on that behavior so that it 
   becomes required for web compat. It would be interesting to know if 
   there are counter-examples, any script-visible behavior that is 
   allowed to vary greatly between implementations without causing 
   scripts to break.
  
  Images aren't required to load at all. Scripts aren't required to run 
  at all. The window size is allowed to be any dimension at all. CSS 
  isn't required to be supported at all. Users are allowed to apply 
  arbitrary user style sheets. Users are allowed to interact with form 
  controls by using the keyboard or the mouse or any other input device.
  
  All of these do break some pages.
 
 That CSS is optional and that users are allowed to apply user style 
 sheets didn't stop you from specifying the Rendering section in great 
 detail.

Optional detail. UAs aren't required to follow that section.


 Making video behavior underdefined just because users should be able 
 to disable video loading in preferences just means that in a few years 
 the behavior of the market leader needs to be reverse engineered and 
 implemented by everyone else.

I do not understand how this particular feature could end up in that 
state any more than the other features I list above.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video element not ready for prime time

[Philip Jägenstedt]

On Thu, 07 Jun 2012 04:06:10 +0200, Kit Grose k...@iqmultimedia.com.au  
wrote:



On 06/06/2012, at 7:44 AM, Ian Hickson wrote:

On Fri, 13 Jan 2012, Kit Grose wrote:


I'd argue that while we did receive in WebM a common codec it does  
not

enjoy the sort of universal adoption required to be able to mandate its
support in the spec, so on that logic, I think establishing a
declarative fallback mechanism is probably required to prevent a
situation where you cannot write a robust HTML5 page with video and
without resorting to JS.


I don't think it's time to give up yet, but maybe I'm overly  
optimistic...



Is there any reason why it wouldn't be prudent to render the content of  
the video element as HTML if the video cannot be played, given that  
fallback content in the video element is already supported for legacy  
browsers in the spec:


Content may be provided inside the video element. User agents should  
not show this content to the user; it is intended for older Web  
browsers which do not support video, so that legacy video plugins can  
be tried, or to show text to the users of these older browsers  
informing them of how to access the video contents.


How are legacy UAs without video support appreciably different from a  
UA with restrictive or limited video support? Surely the author's  
intent in either case is delivering the content in a different way or  
delivering appropriate alternate content.


Even if we eventually get a common codec (which I—perhaps overly  
pessimistically—feel is unlikely), authors are already using the video  
element without supplying whatever that eventual codec will be, and  
users are suffering without reasonable fallbacks.


As it stands, it's almost better (and certainly easier) for authors to  
default to Flash video and use the existing, declarative fallback  
behaviour of the object element to use a video element instead. That  
can't be in the best interest of the open web.


This was discussed in the thread HTML 5 video tag questions in 2009:

http://lists.w3.org/Archives/Public/public-whatwg-archive/2009Jul/thread.html#msg278

The resource selection algorithm never fails - it simply waits for another  
source to be inserted - so the the hard part is when to show the fallback  
content. At the time I was very skeptical of switching back and forth  
between fallback and trying to load a video as more source elements are  
added, and I still am.


--
Philip Jägenstedt
Core Developer
Opera Software

Re: [whatwg] video element not ready for prime time

[Kit Grose]

On 06/06/2012, at 7:44 AM, Ian Hickson wrote:
 On Fri, 13 Jan 2012, Kit Grose wrote:
 
 I'd argue that while we did receive in WebM a common codec it does not 
 enjoy the sort of universal adoption required to be able to mandate its 
 support in the spec, so on that logic, I think establishing a 
 declarative fallback mechanism is probably required to prevent a 
 situation where you cannot write a robust HTML5 page with video and 
 without resorting to JS.
 
 I don't think it's time to give up yet, but maybe I'm overly optimistic...


Is there any reason why it wouldn't be prudent to render the content of the 
video element as HTML if the video cannot be played, given that fallback 
content in the video element is already supported for legacy browsers in the 
spec:

 Content may be provided inside the video element. User agents should not show 
 this content to the user; it is intended for older Web browsers which do not 
 support video, so that legacy video plugins can be tried, or to show text to 
 the users of these older browsers informing them of how to access the video 
 contents.

How are legacy UAs without video support appreciably different from a UA with 
restrictive or limited video support? Surely the author's intent in either 
case is delivering the content in a different way or delivering appropriate 
alternate content.

Even if we eventually get a common codec (which I—perhaps overly 
pessimistically—feel is unlikely), authors are already using the video 
element without supplying whatever that eventual codec will be, and users are 
suffering without reasonable fallbacks.

As it stands, it's almost better (and certainly easier) for authors to default 
to Flash video and use the existing, declarative fallback behaviour of the 
object element to use a video element instead. That can't be in the best 
interest of the open web.

—Kit Grose

Re: [whatwg] video element not ready for prime time

[Chris Double]

On Fri, Jan 13, 2012 at 6:46 AM, Francis Boumphrey
boumphre...@gmail.com wrote:
 Firstly if I use a video with the src attribute

 e.g. video src='myvideo.mp4' controls

 and my user agent does not support the format, all I get (in my versions of
 Opera and Firefox) is a blank screen.

Recent versions of Firefox display a message for the user if the mime
type of the video is not supported instead of a blank screen.

-- 
http://www.bluishcoder.co.nz

Re: [whatwg] video and audio tags need to have volume attribute.

[Ian Hickson]

On Mon, 14 Nov 2011, crocket wrote:

 I know that it's possible to manipulate the volume attribute with
 javascript.
 Many internet forums allow HTML tags but prohibit javascripts, however.
 
 If I was able to set the initial volume with volume attribute of those 
 tags, I would be able to set the volume without javascript.

What's the use case for setting the volume to anything but 1.0 (normal 
volume, controlled by the user's master volume control) or 0.0 (muted)?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video element not ready for prime time

[Ian Hickson]

On Thu, 12 Jan 2012, Francis Boumphrey wrote:

 A few comments about the video element from the point of view of an 
 HTML author. Currently it is not particularly useful, and there is 
 nothing to encourage me to use it rather than embed
 
 Firstly if I use a video with the src attribute
 
 e.g. video src='myvideo.mp4' controls
 
 and my user agent does not support the format, all I get (in my versions 
 of Opera and Firefox) is a blank screen. No message (as I would get with 
 embed) and as far as I can see there is no way for me as an author to 
 know that the video is not being played so I cannot code a 'write 
 around'.

Right, the spec is written with the assumption that we will eventually 
have a single standard codec.


 The option is to make several formats of the same video, e.g. 
 myvideo.mp3, myvideo.mp4, myvideo.ogg etc. and place them in child 
 source elements in the hope that one of the formats will be displayed.
 
 Even here I have a problem. In which order does the user-agent check the 
 source files (in Chrome it seems to be in the order in which they are 
 written, but there is no guidance here in the spec.

The exact algorithm is described in detail. Essentially it's source 
order, yes.


 Also will my user agent down-load the file that it cannot play, thus 
 using up band-width?

Enough of it to determine it cannot play it, yes, unless you provide 
attributes on source for it to determine that it doesn't know the 
format.

Cheers,
-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video element not ready for prime time

[Ian Hickson]

On Fri, 13 Jan 2012, Kit Grose wrote:
 On 13/01/2012, at 4:46 AM, Francis Boumphrey wrote:
  
  Firstly if I use a video with the src attribute
  
  e.g. video src='myvideo.mp4' controls
  
  and my user agent does not support the format, all I get (in my 
  versions of Opera and Firefox) is a blank screen. No message (as I 
  would get with embed) and as far as I can see there is no way for me 
  as an author to know that the video is not being played so I cannot 
  code a 'write around'.
 
 I brought up this same concern in Dec 2009 and Hixie seemed to concur:
 
 On 9/02/2010, Ian Hickson wrote:
  
  Indeed, if we can't get a common codec, the spec as written today is 
  not a particularly good design. If we really can't solve this problem, 
  then we'll have to introduce a declarative way of saying if you can't 
  play any of the videos, here's what I want you to do instead -- but 
  hopefully we won't have to go there.
 
 (see 
 http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2010-February/025028.html)
 
 I'd argue that while we did receive in WebM a common codec it does not 
 enjoy the sort of universal adoption required to be able to mandate its 
 support in the spec, so on that logic, I think establishing a 
 declarative fallback mechanism is probably required to prevent a 
 situation where you cannot write a robust HTML5 page with video and 
 without resorting to JS.

I don't think it's time to give up yet, but maybe I'm overly optimistic...

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video preload implementation feedback

[Simon Pieters]

On Tue, 08 May 2012 18:59:29 +0200, Ian Hickson i...@hixie.ch wrote:


On Thu, 18 Aug 2011, Philip Jägenstedt wrote:


This is true, but as long as a few big browsers implement e.g.
preload=none in a somewhat compatible way, it's hard to imagine page
authors not coming to depend on that behavior so that it becomes
required for web compat. It would be interesting to know if there are
counter-examples, any script-visible behavior that is allowed to vary
greatly between implementations without causing scripts to break.


Images aren't required to load at all. Scripts aren't required to run at
all. The window size is allowed to be any dimension at all. CSS isn't
required to be supported at all. Users are allowed to apply arbitrary
user style sheets. Users are allowed to interact with form controls by
using the keyboard or the mouse or any other input device.

All of these do break some pages.


That CSS is optional and that users are allowed to apply user style sheets  
didn't stop you from specifying the Rendering section in great detail.


Making video behavior underdefined just because users should be able to  
disable video loading in preferences just means that in a few years the  
behavior of the market leader needs to be reverse engineered and  
implemented by everyone else.


--
Simon Pieters
Opera Software

Re: [whatwg] Video- and audio-controls without scripting

[Ian Hickson]

On Sun, 14 Aug 2011, Timo Beermann wrote:

 The new video- and audio-tag are great, but the controls (play/pause, 
 skip forward, skip back, volume, progress bar, time) should be possible 
 without scripting. Some standard-controls that also can be modified with 
 CSS. Because some users deactivate Scripting (for security or whatever 
 other reason) and on other computers (school, university, work,...) you 
 are not able to change the settings, even if you want to. E.g. I use 
 NoScript and only allow scripting on very few trusted sites, that really 
 need it.

On Sun, 14 Aug 2011, Ashley Sheridan wrote:
 
 There are controls available without using javascript. Just add the 
 'controls' attribute to the audio or video tag.

Indeed.

In fact with JS disabled the specification requires that these controls be 
made available. See the second paragraph of:

   http://www.whatwg.org/specs/web-apps/current-work/#user-interface

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video preload implementation feedback

[Ian Hickson]

On Wed, 17 Aug 2011, Philip Jägenstedt wrote:
 
 I'd very much like to see feedback from other implementors. Are you 
 happy with treating autoplay and preload as just hints as in [4] or do 
 you think that we should specify them in greater detail? (This does not 
 preclude having user preferences to override the standardized defaults.)

What _makes_ these attributes just hints _is_ that you can have user 
preferences that override the defaults.


On Thu, 18 Aug 2011, Chris Pearce wrote:
 
 I think autoplay should not be treated as a hint, else it's can't be 
 relied upon to work, and thus would be completely useless.

I think it's imperative that users be able to disable all video playback. 
By definition, that means that not all videos are going to play. This 
means that autoplay can't be relied upon to work.

I don't think that's a problem. There's plenty of examples of things like 
that in the spec. For example, several browsers don't render images at 
all except when requested to render them, and then only in a separate 
window, not inline. I would expect similar behaviour for video.

The great thing about HTML is specifically that it is media-independent in 
this very manner, allowing different user agents to be agents of the user, 
not of the author, displaying the content in the manner most appropriate 
for the user.


On Thu, 18 Aug 2011, Philip Jägenstedt wrote:
 
 I think that too much variation in how preload is implemented is also 
 likely to give compat problems. In 
 http://www.w3.org/Bugs/Public/show_bug.cgi?id=12596#c7 I have an 
 example of what might break when pages inevitably assume that 
 preload=none causes the loadedmetadata event to not be fired.

I do not think that example is realistic, as discussed in the bug.


On Fri, 19 Aug 2011, Chris Pearce wrote:
 On Thu, 18 Aug 2011, Philip Jägenstedt wrote:
  If you only allow the internal state to increase, don't you need to 
  reset it at some point as well? Or is it impossible in your 
  implementation to use preload=auto on one load and 
  preload=metadata on the next due to this?
 
 Oops, that is impossible in our implementation. That's a bug! I'll fix 
 that, thanks for pointing this out. I agree that it we should specify 
 when the preload internal state is updated to prevent this bug in other 
 implementations. Resetting the internal preload state inside the 
 synchronous section of the resource selection algorithm as you suggest 
 is sensible. I agree with you!

There might not _be_ an internal preload state. I don't know how we can 
really specify this.

I'm happy to add a note if that would help avoid bugs; let me know if you 
think that would help (ideally, also let me know what you think the note 
should say).


On Thu, 18 Aug 2011, Philip Jägenstedt wrote:
 
 This is true, but as long as a few big browsers implement e.g. 
 preload=none in a somewhat compatible way, it's hard to imagine page 
 authors not coming to depend on that behavior so that it becomes 
 required for web compat. It would be interesting to know if there are 
 counter-examples, any script-visible behavior that is allowed to vary 
 greatly between implementations without causing scripts to break.

Images aren't required to load at all. Scripts aren't required to run at 
all. The window size is allowed to be any dimension at all. CSS isn't 
required to be supported at all. Users are allowed to apply arbitrary 
user style sheets. Users are allowed to interact with form controls by 
using the keyboard or the mouse or any other input device.

All of these do break some pages.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] video element not ready for prime time

[Philip Jägenstedt]

On Fri, 13 Jan 2012 01:53:23 +0100, Aryeh Gregor a...@aryeh.name wrote:


On Thu, Jan 12, 2012 at 12:46 PM, Francis Boumphrey
boumphre...@gmail.com wrote:

e.g. video src='myvideo.mp4' controls

and my user agent does not support the format, all I get (in my  
versions of

Opera and Firefox) is a blank screen. No message (as I would get with
embed) and as far as I can see there is no way for me as an author to
know that the video is not being played so I cannot code a 'write  
around'.


Boris answered the rest of your questions, but here's a way to detect  
errors:


data:text/html,!doctype html
video src='nonexistent.mp4' controls onerror='alert(error)'/video

If you use source, the error event is fired at the source element
instead and doesn't bubble (why not?), so you have to put onerror on
the source element instead.


Or use a capturing event listener.


I don't know why UAs don't provide their own error messages, though.
They provide error icons for failed image loads.


I agree that we should do something, but it would be limited to video  
controls as audio doesn't have a rendering area at all and for video  
without controls it could interfere with the scripted interface. For  
audio controls you could always make the controls look dead of course,  
to make it more clear that there was a problem.


--
Philip Jägenstedt
Core Developer
Opera Software

Re: [whatwg] video element not ready for prime time

[Boris Zbarsky]

On 1/12/12 12:46 PM, Francis Boumphrey wrote:

and as far as I can see there is no way for me as an author to
know that the video is not being played


If true, this should be simple to fix.  But I would think that there was 
a way to detect this via the readyState or error properties.  In 
particular, if the error.code is MEDIA_ERR_SRC_NOT_SUPPORTED that would 
seem to be a strong indication that the format could not be played, right?



Even here I have a problem. In which order does the user-agent check the
source files (in Chrome it seems to be in the order in which they are
written, but there is no guidance here in the spec.


http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#concept-media-load-algorithm 
step 3 says:


  Otherwise, if the media element does not have a src attribute but has
  a source element child, then let mode be children and let candidate
  be the first such source element child in tree order.

and same section starting at Otherwise, the source elements will be 
used; run these substeps: explicitly talks about trying the source 
elements in document order.


So what Chrome is doing is exactly what the spec requires.


Also will my user agent down-load the file that it cannot play, thus using
up band-width?


That's up to the to some extent UA, but I'd think that if you list a 
type=whatever and the UA doesn't support the type, any sane UA would 
skip the download.  Specifically, see substep 5 of the run these 
substeps section mentioned above:


  5. If candidate has a type attribute whose value, when parsed as a
  MIME type (including any codecs described by the codecs parameter,
  for types that define that parameter), represents a type that the
  user agent knows it cannot render, then end the synchronous section,
  and jump down to the failed step below.

So the only way for a UA to not skip the source is if it doesn't know 
it can't render the type.


-Boris

Re: [whatwg] video element not ready for prime time

[Ami Fischman]

On Thu, Jan 12, 2012 at 9:46 AM, Francis Boumphrey boumphre...@gmail.comwrote:

 In which order does the user-agent check the
 source files (in Chrome it seems to be in the order in which they are
 written, but there is no guidance here in the spec.


The spec does specify this, and chrome fails to follow the spec.
See http://webk.it/75154 for details.

Cheers,
-a

Re: [whatwg] video element not ready for prime time

[Kit Grose]

On 13/01/2012, at 4:46 AM, Francis Boumphrey wrote:

 Firstly if I use a video with the src attribute
 
 e.g. video src='myvideo.mp4' controls
 
 and my user agent does not support the format, all I get (in my versions of
 Opera and Firefox) is a blank screen. No message (as I would get with
 embed) and as far as I can see there is no way for me as an author to
 know that the video is not being played so I cannot code a 'write around'.

I brought up this same concern in Dec 2009 and Hixie seemed to concur:

On 9/02/2010, Ian Hickson wrote:

 Indeed, if we can't get a common codec, the spec as written today is not 
 a particularly good design. If we really can't solve this problem, then 
 we'll have to introduce a declarative way of saying if you can't play any 
 of the videos, here's what I want you to do instead -- but hopefully we 
 won't have to go there.

(see 
http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2010-February/025028.html)

I'd argue that while we did receive in WebM a common codec it does not enjoy 
the sort of universal adoption required to be able to mandate its support in 
the spec, so on that logic, I think establishing a declarative fallback 
mechanism is probably required to prevent a situation where you cannot write a 
robust HTML5 page with video and without resorting to JS.

—Kit Grose

Re: [whatwg] video element not ready for prime time

[Aryeh Gregor]

On Thu, Jan 12, 2012 at 12:46 PM, Francis Boumphrey
boumphre...@gmail.com wrote:
 e.g. video src='myvideo.mp4' controls

 and my user agent does not support the format, all I get (in my versions of
 Opera and Firefox) is a blank screen. No message (as I would get with
 embed) and as far as I can see there is no way for me as an author to
 know that the video is not being played so I cannot code a 'write around'.

Boris answered the rest of your questions, but here's a way to detect errors:

data:text/html,!doctype html
video src='nonexistent.mp4' controls onerror='alert(error)'/video

If you use source, the error event is fired at the source element
instead and doesn't bubble (why not?), so you have to put onerror on
the source element instead.

I don't know why UAs don't provide their own error messages, though.
They provide error icons for failed image loads.

Re: [whatwg] video preload implementation feedback

[Philip Jägenstedt]

On Thu, 18 Aug 2011 04:50:17 +0200, Chris Pearce cpea...@mozilla.com  
wrote:



I implemented preload support in Firefox.


On 18/08/2011 3:44 a.m., Philip Jägenstedt wrote:
I'd very much like to see feedback from other implementors. Are you  
happy with treating autoplay and preload as just hints as in [4] or  
do you think that we should specify them in greater detail?


I think autoplay should not be treated as a hint, else it's can't be  
relied upon to work, and thus would be completely useless.


Preload is a less critical; if it's not supported users will just end up  
loading more data, which isn't too bad. Clients that care more about  
bandwidth will probably be more likely to support it.


I think that too much variation in how preload is implemented is also  
likely to give compat problems. In  
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12596#c7 I have an example  
of what might break when pages inevitably assume that preload=none  
causes the loadedmetadata event to not be fired.



== Resetting internal preload state ==

Due to the above, it's necessary to reset the internal preload state at  
some point. Otherwise, a script like this wouldn't work:


function setSource(url) {
  var v = document.querySelector('video');
  v.src = url;
  v.preload = none;
  v.onplay = function() { v.preload = autoplay; };


Did you mean |v.preload = auto;| here instead? or |v.autoplay =  
true;|? It seems in this case the onplay handler would only happen if  
the user pressed play on the controls or from script, so the preload  
action be promoted to auto anyway since the resource is playing. I guess  
that's what you're getting at with your point about preload internal  
state promotion?


Oops, it should be v.preload = auto. Indeed this example is more  
complicated than necessary. Promotion of the internal state due to user  
interaction also needs to be reset in the resource selection algorithm, so  
any time that a HTMLMediaElement is reused this is a problem.



}

If a previous resource was playing and preload was set to autoplay by  
script, then we still want preload=none to apply to the new resource.  
To solve this, we are resetting the internal preload state as part of  
the resource selection algorithm, right before step 5 to fire the  
loadstart event. There are various other places one could do this, but  
we think it is important to do it in the async section so that the  
order of setting .src and .preload does not matter.


Currently we update the internal preload action whenever the value of  
the preload state changes, and we check it's not preload=none before  
kicking off a new resource load (resource fetch algorithm step 2) and we  
check it again when we reach loadedmetadata and suspend the load if it's  
preload=metadata.


I think the preload=metadata case is implied by the spec, but having it  
explicitly stated wouldn't hurt.


It sounds like we're applying preload=none at exactly the same point,  
that's good!


If you only allow the internal state to increase, don't you need to reset  
it at some point as well? Or is it impossible in your implementation to  
use preload=auto on one load and preload=metadata on the next due to  
this?



== video preload=none ==

It's not possible to specify exactly how much preload=metadata and  
preload=auto buffers and when, but this is possible for  
preload=none. This is what we do:


After step 1 of the source selection algorithm, if preload==none, set  
networkState to IDLE, fire a suspend event and set the  
delaying-the-load-event flag to false.


This is actually specified now in step 2 of the resource fetch algorithm  
(this must have been added after we implemented @preload).


Right, this is the partially accepted resolution of  
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12596


Doing this at step 1 of the resource selection algorithm means that if  
you're loading from child source elements, and none of them have a  
supported type, then you won't report the error until something calls  
play() or load() explicitly.


Oops, I meant resource fetch algorithm and what I have implemented is  
exactly what the spec says, except that it's unconditional and the  
implementation-defined event is playing or seeking.


Thanks for your feedback!

--
Philip Jägenstedt
Core Developer
Opera Software

Re: [whatwg] video preload implementation feedback

[Philip Jägenstedt]

On Thu, 18 Aug 2011 05:36:42 +0200, Bjartur Thorlacius  
svartma...@gmail.com wrote:



Þann mið 17.ágú 2011 15:44, skrifaði Philip Jägenstedt:

I'd very much like to see feedback from other implementors. Are you
happy with treating autoplay and preload as just hints as in [4] or do
you think that we should specify them in greater detail? (This does not
preclude having user preferences to override the standardized defaults.)

If an UA may ignore them, than them being honored can't be relied upon -  
rendering the default configuration a question of implementation (i.e.  
providing sane defaults for the user agent's target user base) but not  
of interoperability.


This is true, but as long as a few big browsers implement e.g.  
preload=none in a somewhat compatible way, it's hard to imagine page  
authors not coming to depend on that behavior so that it becomes required  
for web compat. It would be interesting to know if there are  
counter-examples, any script-visible behavior that is allowed to vary  
greatly between implementations without causing scripts to break.


--
Philip Jägenstedt
Core Developer
Opera Software

Re: [whatwg] video preload implementation feedback

[Chris Pearce]

On 19/08/2011 12:01 a.m., Philip Jägenstedt wrote:
I think that too much variation in how preload is implemented is also 
likely to give compat problems. In 
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12596#c7 I have an 
example of what might break when pages inevitably assume that 
preload=none causes the loadedmetadata event to not be fired.


This seems a valid reason to change from should to must for preload. 
I agree.


If you only allow the internal state to increase, don't you need to 
reset it at some point as well? Or is it impossible in your 
implementation to use preload=auto on one load and 
preload=metadata on the next due to this?


Oops, that is impossible in our implementation. That's a bug! I'll fix 
that, thanks for pointing this out. I agree that it we should specify 
when the preload internal state is updated to prevent this bug in other 
implementations. Resetting the internal preload state inside the 
synchronous section of the resource selection algorithm as you suggest 
is sensible. I agree with you!



Cheers,
Chris Pearce.

Re: [whatwg] video preload implementation feedback

[Chris Pearce]

I implemented preload support in Firefox.


On 18/08/2011 3:44 a.m., Philip Jägenstedt wrote:
I'd very much like to see feedback from other implementors. Are you 
happy with treating autoplay and preload as just hints as in [4] or 
do you think that we should specify them in greater detail?


I think autoplay should not be treated as a hint, else it's can't be 
relied upon to work, and thus would be completely useless.


Preload is a less critical; if it's not supported users will just end up 
loading more data, which isn't too bad. Clients that care more about 
bandwidth will probably be more likely to support it.




== Dynamically changing preload ==

It makes no sense for a script to change preload=auto to 
preload=none. Going from preload=auto to preload=metadata isn't 
nonsensical, but supporting it would allow authors to toggle it 
continuously to work around buggy buffering behavior. I'd much rather 
that buffering problems be fixed in the browser, so I don't want to 
support this. Consequently, we only allow the internal preload states 
to increase, not decrease. I understand that Mozilla has done the 
same. Unless there are strong reasons not do, I think this should be 
spec'd.


I agree.



== Resetting internal preload state ==

Due to the above, it's necessary to reset the internal preload state 
at some point. Otherwise, a script like this wouldn't work:


function setSource(url) {
  var v = document.querySelector('video');
  v.src = url;
  v.preload = none;
  v.onplay = function() { v.preload = autoplay; };


Did you mean |v.preload = auto;| here instead? or |v.autoplay = 
true;|? It seems in this case the onplay handler would only happen if 
the user pressed play on the controls or from script, so the preload 
action be promoted to auto anyway since the resource is playing. I guess 
that's what you're getting at with your point about preload internal 
state promotion?



}

If a previous resource was playing and preload was set to autoplay 
by script, then we still want preload=none to apply to the new 
resource. To solve this, we are resetting the internal preload state 
as part of the resource selection algorithm, right before step 5 to 
fire the loadstart event. There are various other places one could do 
this, but we think it is important to do it in the async section so 
that the order of setting .src and .preload does not matter.


Currently we update the internal preload action whenever the value of 
the preload state changes, and we check it's not preload=none before 
kicking off a new resource load (resource fetch algorithm step 2) and we 
check it again when we reach loadedmetadata and suspend the load if it's 
preload=metadata.


I think the preload=metadata case is implied by the spec, but having it 
explicitly stated wouldn't hurt.




== video preload=none ==

It's not possible to specify exactly how much preload=metadata and 
preload=auto buffers and when, but this is possible for 
preload=none. This is what we do:


After step 1 of the source selection algorithm, if preload==none, 
set networkState to IDLE, fire a suspend event and set the 
delaying-the-load-event flag to false.


This is actually specified now in step 2 of the resource fetch algorithm 
(this must have been added after we implemented @preload).


Doing this at step 1 of the resource selection algorithm means that if 
you're loading from child source elements, and none of them have a 
supported type, then you won't report the error until something calls 
play() or load() explicitly.



Regards,
Chris Pearce.

Re: [whatwg] video preload implementation feedback

[Bjartur Thorlacius]

Þann mið 17.ágú 2011 15:44, skrifaði Philip Jägenstedt:

I'd very much like to see feedback from other implementors. Are you
happy with treating autoplay and preload as just hints as in [4] or do
you think that we should specify them in greater detail? (This does not
preclude having user preferences to override the standardized defaults.)

If an UA may ignore them, than them being honored can't be relied upon - 
rendering the default configuration a question of implementation (i.e. 
providing sane defaults for the user agent's target user base) but not 
of interoperability.

Re: [whatwg] Video- and audio-controls without scripting

[Ashley Sheridan]

Timo Beermann timo.beerm...@googlemail.com wrote:

The new video- and audio-tag are great, but the controls (play/pause,
skip forward, skip back, volume, progress bar, time) should be
possible without scripting. Some standard-controls that also can be
modified with CSS. Because some users deactivate Scripting (for
security or whatever other reason) and on other computers (school,
university, work,...) you are not able to change the settings, even if
you want to. E.g. I use NoScript and only allow scripting on very few
trusted sites, that really need it.

Timo Beermann

There are controls available without using javascript. Just add the 'controls' 
attribute to the audio or video tag. The scriptable elements just allow you to 
create custom controls that can have further actions, or look and behave 
slightly differently. The controls and appearance of them is dependant on the 
browser and platform in use, but generally there are the basic play/pause, 
stop, forward  backward buttons, a seek bar and a volume control.

Thanks,
Ash
http://www.ashleysheridan.co.uk
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: [whatwg] Video- and audio-controls without scripting

[Kyle Huey]

On Sun, Aug 14, 2011 at 3:33 AM, Timo Beermann timo.beerm...@googlemail.com
 wrote:

 The new video- and audio-tag are great, but the controls (play/pause,
 skip forward, skip back, volume, progress bar, time) should be
 possible without scripting. Some standard-controls that also can be
 modified with CSS. Because some users deactivate Scripting (for
 security or whatever other reason) and on other computers (school,
 university, work,...) you are not able to change the settings, even if
 you want to. E.g. I use NoScript and only allow scripting on very few
 trusted sites, that really need it.


This is a Gecko issue https://bugzilla.mozilla.org/show_bug.cgi?id=449358,
and has nothing to do with the spec.

- Kyle

Re: [whatwg] Video feedback

[Ian Hickson]

On Thu, 7 Jul 2011, Eric Winkelman wrote:
 On Thursday, June 02 Ian Hickson wrote:
  On Fri, 18 Mar 2011, Eric Winkelman wrote:
  
   For in-band metadata tracks, there is neither a standard way to 
   represent the type of metadata in the HTMLTrackElement interface nor 
   is there a standard way to represent multiple different types of 
   metadata tracks.
  
  There can be a standard way. The idea is that all the types of 
  metadata tracks that browsers will support should be specified so that 
  all browsers can map them the same way. I'm happy to work with anyone 
  interested in writing such a mapping spec, just let me know.
 
 I would be very interested in working on this spec.

It would be several specs, probably, each focusing on a particular set of 
metadata in a particular format (e.g. advertising timings in an MPEG 
wrapper, or whatever).


 What's the next step?

First, research: what formats and metadata streams are you interested in? 
Who uses them? How are they implemented in producers and (more 
importantly) consumers today? What are the use cases?

Second, describe the problem: make a clear statement of purpose that 
scopes the effort to provide guidelines to prevent feature creep.

Third, listen to implementors: find those that are interested in 
implementing this particular mapping of metadata to the DOM API, get their 
input, see what they want.

Fourth, implement: make or have someone else make an experimental 
implementation of a mapping that addresses the problem described in the 
earlier steps.

Fifth, specify: write a specification that describes the mapping described 
in step two, based on what you've researched in step one and based on the 
feedback from steps three and four.

Sixth, test: update the experimental implement to fit the spec, get other 
implementations to implement the spec. Have real users play with it.

Seventh, simplify: remove what you don't need.

Finally, iterate: repeat all these steps for as long as there's any 
interest in this mapping, fixing problems, adding new features if they're 
needed, removing old features that didn't get used or implemented, etc.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Video feedback

[Bob Lund]

 -Original Message-
 From: whatwg-boun...@lists.whatwg.org [mailto:whatwg-
 boun...@lists.whatwg.org] On Behalf Of Mark Watson
 Sent: Monday, June 20, 2011 2:29 AM
 To: Eric Carlson
 Cc: Silvia Pfeiffer; whatwg Group; Simon Pieters
 Subject: Re: [whatwg] Video feedback
 
 
 On Jun 9, 2011, at 4:32 PM, Eric Carlson wrote:
 
 
  On Jun 9, 2011, at 12:02 AM, Silvia Pfeiffer wrote:
 
  On Thu, Jun 9, 2011 at 4:34 PM, Simon Pieters sim...@opera.com
 wrote:
  On Thu, 09 Jun 2011 03:47:49 +0200, Silvia Pfeiffer
  silviapfeiff...@gmail.com wrote:
 
  For commercial video providers, the tracks in a live stream change
  all the time; this is not limited to audio and video tracks but
  would include text tracks as well.
 
  OK, all this indicates to me that we probably want a
 metadatachanged
  event to indicate there has been a change and that JS may need to
  check some of its assumptions.
 
  We already have durationchange. Duration is metadata. If we want to
  support changes to width/height, and the script is interested in
  when that happens, maybe there should be a dimensionchange event
  (but what's the use case for changing width/height mid-stream?).
  Does the spec support changes to text tracks mid-stream?
 
  It's not about what the spec supports, but what real-world streams
 provide.
 
  I don't think it makes sense to put an event on every single type of
  metadata that can change. Most of the time, when you have a stream
  change, many variables will change together, so a single event is a
  lot less events to raise. It's an event that signifies that the media
  framework has reset the video/audio decoding pipeline and loaded a
  whole bunch of new stuff. You should imagine it as a concatenation of
  different media resources. And yes, they can have different track
  constitution and different audio sampling rate (which the audio API
  will care about) etc etc.
 
   In addition, it is possible for a stream to lose or gain an audio
 track. In this case the dimensions won't change but a script may want to
 react to the change in audioTracks.
 
 The TrackList object has an onchanged event, which I assumed would fire
 when any of the information in the TrackList changes (e.g. tracks added
 or removed). But actually the spec doesn't state when this event fires
 (as far as I could tell - unless it is implied by some general
 definition of events called onchanged).
 
 Should there be some clarification here ?
 
 
   I agree with Silvia, a more generic metadata changed event makes
 more sense.
 
 Yes, and it should support the case in which text tracks are
 added/removed too.

Has there been a bug submitted to add a metadata changed event when video, 
audio or text tracks are added or deleted from a media resource?

Thanks,
Bob Lund

 
 Also, as Eric (C) pointed out, one of the things which can change is
 which of several available versions of the content is being rendered
 (for adaptive bitrate cases). This doesn't necessarily change any of the
 metadata currently exposed on the video element, but nevertheless it's
 information that the application may need. It would be nice to expose
 some kind of identifier for the currently rendered stream and have an
 event when this changes. I think that a stream-format-supplied
 identifier would be sufficient.
 
 ...Mark
 
 
  eric
 
 

Re: [whatwg] Video feedback

[Eric Winkelman]

On Thursday, June 02 Ian Hickson wrote:

 On Fri, 18 Mar 2011, Eric Winkelman wrote:
 
  For in-band metadata tracks, there is neither a standard way to 
  represent the type of metadata in the HTMLTrackElement interface nor 
  is there a standard way to represent multiple different types of 
  metadata tracks.
 
 There can be a standard way. The idea is that all the types of 
 metadata tracks that browsers will support should be specified so that 
 all browsers can map them the same way. I'm happy to work with anyone 
 interested in writing such a mapping spec, just let me know.

I would be very interested in working on this spec.  

CableLabs works with numerous groups delivering content containing a variety of 
metadata, so we have a good idea what is currently used.  We're also working 
with the groups defining adaptive bit rate delivery protocols about how 
metadata might be carried.

What's the next step?

Eric

Re: [whatwg] Video feedback

[Mark Watson]

On Jun 9, 2011, at 4:32 PM, Eric Carlson wrote:

 
 On Jun 9, 2011, at 12:02 AM, Silvia Pfeiffer wrote:
 
 On Thu, Jun 9, 2011 at 4:34 PM, Simon Pieters sim...@opera.com wrote:
 On Thu, 09 Jun 2011 03:47:49 +0200, Silvia Pfeiffer
 silviapfeiff...@gmail.com wrote:
 
 For commercial video providers, the tracks in a live stream change all
 the time; this is not limited to audio and video tracks but would include
 text tracks as well.
 
 OK, all this indicates to me that we probably want a metadatachanged
 event to indicate there has been a change and that JS may need to
 check some of its assumptions.
 
 We already have durationchange. Duration is metadata. If we want to support
 changes to width/height, and the script is interested in when that happens,
 maybe there should be a dimensionchange event (but what's the use case for
 changing width/height mid-stream?). Does the spec support changes to text
 tracks mid-stream?
 
 It's not about what the spec supports, but what real-world streams provide.
 
 I don't think it makes sense to put an event on every single type of
 metadata that can change. Most of the time, when you have a stream
 change, many variables will change together, so a single event is a
 lot less events to raise. It's an event that signifies that the media
 framework has reset the video/audio decoding pipeline and loaded a
 whole bunch of new stuff. You should imagine it as a concatenation of
 different media resources. And yes, they can have different track
 constitution and different audio sampling rate (which the audio API
 will care about) etc etc.
 
  In addition, it is possible for a stream to lose or gain an audio track. In 
 this case the dimensions won't change but a script may want to react to the 
 change in audioTracks. 

The TrackList object has an onchanged event, which I assumed would fire when 
any of the information in the TrackList changes (e.g. tracks added or removed). 
But actually the spec doesn't state when this event fires (as far as I could 
tell - unless it is implied by some general definition of events called 
onchanged).

Should there be some clarification here ?

 
  I agree with Silvia, a more generic metadata changed event makes more 
 sense. 

Yes, and it should support the case in which text tracks are added/removed too.

Also, as Eric (C) pointed out, one of the things which can change is which of 
several available versions of the content is being rendered (for adaptive 
bitrate cases). This doesn't necessarily change any of the metadata currently 
exposed on the video element, but nevertheless it's information that the 
application may need. It would be nice to expose some kind of identifier for 
the currently rendered stream and have an event when this changes. I think that 
a stream-format-supplied identifier would be sufficient.

...Mark

 
 eric
 
 

Re: [whatwg] Video feedback

[Silvia Pfeiffer]

On Mon, Jun 20, 2011 at 6:29 PM, Mark Watson wats...@netflix.com wrote:

 On Jun 9, 2011, at 4:32 PM, Eric Carlson wrote:


 On Jun 9, 2011, at 12:02 AM, Silvia Pfeiffer wrote:

 On Thu, Jun 9, 2011 at 4:34 PM, Simon Pieters sim...@opera.com wrote:
 On Thu, 09 Jun 2011 03:47:49 +0200, Silvia Pfeiffer
 silviapfeiff...@gmail.com wrote:

 For commercial video providers, the tracks in a live stream change all
 the time; this is not limited to audio and video tracks but would include
 text tracks as well.

 OK, all this indicates to me that we probably want a metadatachanged
 event to indicate there has been a change and that JS may need to
 check some of its assumptions.

 We already have durationchange. Duration is metadata. If we want to support
 changes to width/height, and the script is interested in when that happens,
 maybe there should be a dimensionchange event (but what's the use case for
 changing width/height mid-stream?). Does the spec support changes to text
 tracks mid-stream?

 It's not about what the spec supports, but what real-world streams provide.

 I don't think it makes sense to put an event on every single type of
 metadata that can change. Most of the time, when you have a stream
 change, many variables will change together, so a single event is a
 lot less events to raise. It's an event that signifies that the media
 framework has reset the video/audio decoding pipeline and loaded a
 whole bunch of new stuff. You should imagine it as a concatenation of
 different media resources. And yes, they can have different track
 constitution and different audio sampling rate (which the audio API
 will care about) etc etc.

  In addition, it is possible for a stream to lose or gain an audio track. In 
 this case the dimensions won't change but a script may want to react to the 
 change in audioTracks.

 The TrackList object has an onchanged event, which I assumed would fire when 
 any of the information in the TrackList changes (e.g. tracks added or 
 removed). But actually the spec doesn't state when this event fires (as far 
 as I could tell - unless it is implied by some general definition of events 
 called onchanged).

 Should there be some clarification here ?

I understood that to relate to a change of cues only, since it is on
the tracklist. I.e. it's an aggregate event from the oncuechange event
of a cue inside the track. I didn't think it would relate to a change
of existence of that track.

Note that the even is attached to the TrackList, not the TrackList[],
so it cannot be raised when a track is added or removed, only when
something inside the TrackList changes.


  I agree with Silvia, a more generic metadata changed event makes more 
 sense.

 Yes, and it should support the case in which text tracks are added/removed 
 too.

Yes, it needs to be an event on the MediaElement.


 Also, as Eric (C) pointed out, one of the things which can change is which of 
 several available versions of the content is being rendered (for adaptive 
 bitrate cases). This doesn't necessarily change any of the metadata currently 
 exposed on the video element, but nevertheless it's information that the 
 application may need. It would be nice to expose some kind of identifier for 
 the currently rendered stream and have an event when this changes. I think 
 that a stream-format-supplied identifier would be sufficient.


I don't know about the adaptive streaming situation. I think that is
more about statistics/metrics rather than about change of resource.
All the alternatives in an adaptive streaming resource should
provide the same number of tracks and the same video dimensions, just
at different bitrate/quality, no? Different video dimensions should be
provided through the source element and @media attribute, but within
an adaptive stream, the alternatives should be consistent because the
target device won't change. I guess this is a discussion for another
thread... :-)

Cheers,
Silvia.

Re: [whatwg] Video feedback

[Mark Watson]

On Jun 20, 2011, at 10:42 AM, Silvia Pfeiffer wrote:

On Mon, Jun 20, 2011 at 6:29 PM, Mark Watson 
wats...@netflix.commailto:wats...@netflix.com wrote:

On Jun 9, 2011, at 4:32 PM, Eric Carlson wrote:


On Jun 9, 2011, at 12:02 AM, Silvia Pfeiffer wrote:

On Thu, Jun 9, 2011 at 4:34 PM, Simon Pieters 
sim...@opera.commailto:sim...@opera.com wrote:
On Thu, 09 Jun 2011 03:47:49 +0200, Silvia Pfeiffer
silviapfeiff...@gmail.commailto:silviapfeiff...@gmail.com wrote:

For commercial video providers, the tracks in a live stream change all
the time; this is not limited to audio and video tracks but would include
text tracks as well.

OK, all this indicates to me that we probably want a metadatachanged
event to indicate there has been a change and that JS may need to
check some of its assumptions.

We already have durationchange. Duration is metadata. If we want to support
changes to width/height, and the script is interested in when that happens,
maybe there should be a dimensionchange event (but what's the use case for
changing width/height mid-stream?). Does the spec support changes to text
tracks mid-stream?

It's not about what the spec supports, but what real-world streams provide.

I don't think it makes sense to put an event on every single type of
metadata that can change. Most of the time, when you have a stream
change, many variables will change together, so a single event is a
lot less events to raise. It's an event that signifies that the media
framework has reset the video/audio decoding pipeline and loaded a
whole bunch of new stuff. You should imagine it as a concatenation of
different media resources. And yes, they can have different track
constitution and different audio sampling rate (which the audio API
will care about) etc etc.

 In addition, it is possible for a stream to lose or gain an audio track. In 
this case the dimensions won't change but a script may want to react to the 
change in audioTracks.

The TrackList object has an onchanged event, which I assumed would fire when 
any of the information in the TrackList changes (e.g. tracks added or removed). 
But actually the spec doesn't state when this event fires (as far as I could 
tell - unless it is implied by some general definition of events called 
onchanged).

Should there be some clarification here ?

I understood that to relate to a change of cues only, since it is on
the tracklist. I.e. it's an aggregate event from the oncuechange event
of a cue inside the track. I didn't think it would relate to a change
of existence of that track.

Note that the even is attached to the TrackList, not the TrackList[],
so it cannot be raised when a track is added or removed, only when
something inside the TrackList changes.

Are we talking about the same thing ? There is no TrackList array and TrackList 
is only used for audio/video, not text, so I don't understand the comment about 
cues.

I'm talking about 
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist
 which is the base class for MultipleTrackList and ExclusiveTrackList used to 
represent all the audio and video tracks (respectively). One instance of the 
object represents all the tracks, so I would assume that a change in the number 
of tracks is a change to this object.




 I agree with Silvia, a more generic metadata changed event makes more sense.

Yes, and it should support the case in which text tracks are added/removed too.

Yes, it needs to be an event on the MediaElement.


Also, as Eric (C) pointed out, one of the things which can change is which of 
seve
ral available versions of the content is being rendered (for adaptive bitrate 
cases). This doesn't necessarily change any of the metadata currently exposed 
on the video element, but nevertheless it's information that the application 
may need. It would be nice to expose some kind of identifier for the currently 
rendered stream and have an event when this changes. I think that a 
stream-format-supplied identifier would be sufficient.


I don't know about the adaptive streaming situation. I think that is
more about statistics/metrics rather than about change of resource.
All the alternatives in an adaptive streaming resource should
provide the same number of tracks and the same video dimensions, just
at different bitrate/quality, no?

I think of the different adaptive versions on a per-track basis (i.e. the 
alternatives are *within* each track), not a bunch of alternatives each of 
which contains several tracks. Both are possible, of course.

It's certainly possible (indeed common) for different bitrate video encodings 
to have different resolutions - there are video encoding reasons to do this. Of 
course the aspect ratio should not change and nor should the dimensions on the 
screen (both would be a little peculiar for the user).

Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not the 
same as the resolution (for a start, they are in CSS pixels, which 

Re: [whatwg] Video feedback

[Silvia Pfeiffer]

On Mon, Jun 20, 2011 at 7:31 PM, Mark Watson wats...@netflix.com wrote:

 The TrackList object has an onchanged event, which I assumed would fire when
 any of the information in the TrackList changes (e.g. tracks added or
 removed). But actually the spec doesn't state when this event fires (as far
 as I could tell - unless it is implied by some general definition of events
 called onchanged).

 Should there be some clarification here ?

 I understood that to relate to a change of cues only, since it is on
 the tracklist. I.e. it's an aggregate event from the oncuechange event
 of a cue inside the track. I didn't think it would relate to a change
 of existence of that track.

 Note that the even is attached to the TrackList, not the TrackList[],
 so it cannot be raised when a track is added or removed, only when
 something inside the TrackList changes.

 Are we talking about the same thing ? There is no TrackList array and
 TrackList is only used for audio/video, not text, so I don't understand the
 comment about cues.
 I'm talking
 about http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist which
 is the base class for MultipleTrackList and ExclusiveTrackList used to
 represent all the audio and video tracks (respectively). One instance of the
 object represents all the tracks, so I would assume that a change in the
 number of tracks is a change to this object.

Ah yes, you're right: I got confused.

It says Whenever the selected track is changed, the user agent must
queue a task to fire a simple event named change at the
MultipleTrackList object. This means it fires when the selectedIndex
is changed, i.e. the user chooses a different track for rendering. I
still don't think it relates to changes in the composition of tracks
of a resource. That should be something different and should probably
be on the MediaElement and not on the track list to also cover changes
in text tracks.


 Also, as Eric (C) pointed out, one of the things which can change is which
 of several available versions of the content is being rendered (for adaptive
 bitrate cases). This doesn't necessarily change any of the metadata
 currently exposed on the video element, but nevertheless it's information
 that the application may need. It would be nice to expose some kind of
 identifier for the currently rendered stream and have an event when this
 changes. I think that a stream-format-supplied identifier would be
 sufficient.

 I don't know about the adaptive streaming situation. I think that is
 more about statistics/metrics rather than about change of resource.
 All the alternatives in an adaptive streaming resource should
 provide the same number of tracks and the same video dimensions, just
 at different bitrate/quality, no?

 I think of the different adaptive versions on a per-track basis (i.e. the
 alternatives are *within* each track), not a bunch of alternatives each of
 which contains several tracks. Both are possible, of course.

 It's certainly possible (indeed common) for different bitrate video
 encodings to have different resolutions - there are video encoding reasons
 to do this. Of course the aspect ratio should not change and nor should the
 dimensions on the screen (both would be a little peculiar for the user).

 Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not
 the same as the resolution (for a start, they are in CSS pixels, which are
 square), but I think it quite likely that if the resolution of the video
 changes than the videoWidth and videoHeight might change. I'd be interested
 to hear how existing implementations relate resolution to videoWidth and
 videoHeight.

Well, if videoWidth and videoHeight change and no dimensions on the
video are provided through CSS, then surely the video will change size
and the display will shrink. That would be a terrible user experience.
For that reason I would suggest that such a change not be made in
alternative adaptive streams.


 Different video dimensions should be
 provided through the source element and @media attribute, but within
 an adaptive stream, the alternatives should be consistent because the
 target device won't change. I guess this is a discussion for another
 thread... :-)

 Possibly ;-) The device knows much better than the page author what
 capabilities it has and so what resolutions are suitable for the device. So
 it is better to provide all the alternatives as a single resource and have
 the device work out which subset it can support. Or at least, the list
 should be provided all at the same level - there is no rationale for a
 hierarchy of alternatives.

The way in which HTML deals with different devices and their different
capabilities is through media queries. As a author you provide your
content with different versions of media-dependent style sheets and
content, so that when you view the page with a different device, the
capabilities of the device select the right style sheet and 

Re: [whatwg] Video feedback

[Mark Watson]

On Jun 20, 2011, at 11:52 AM, Silvia Pfeiffer wrote:

 On Mon, Jun 20, 2011 at 7:31 PM, Mark Watson wats...@netflix.com wrote:
 
 The TrackList object has an onchanged event, which I assumed would fire 
 when
 any of the information in the TrackList changes (e.g. tracks added or
 removed). But actually the spec doesn't state when this event fires (as far
 as I could tell - unless it is implied by some general definition of events
 called onchanged).
 
 Should there be some clarification here ?
 
 I understood that to relate to a change of cues only, since it is on
 the tracklist. I.e. it's an aggregate event from the oncuechange event
 of a cue inside the track. I didn't think it would relate to a change
 of existence of that track.
 
 Note that the even is attached to the TrackList, not the TrackList[],
 so it cannot be raised when a track is added or removed, only when
 something inside the TrackList changes.
 
 Are we talking about the same thing ? There is no TrackList array and
 TrackList is only used for audio/video, not text, so I don't understand the
 comment about cues.
 I'm talking
 about 
 http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist
  which
 is the base class for MultipleTrackList and ExclusiveTrackList used to
 represent all the audio and video tracks (respectively). One instance of the
 object represents all the tracks, so I would assume that a change in the
 number of tracks is a change to this object.
 
 Ah yes, you're right: I got confused.
 
 It says Whenever the selected track is changed, the user agent must
 queue a task to fire a simple event named change at the
 MultipleTrackList object. This means it fires when the selectedIndex
 is changed, i.e. the user chooses a different track for rendering. I
 still don't think it relates to changes in the composition of tracks
 of a resource. That should be something different and should probably
 be on the MediaElement and not on the track list to also cover changes
 in text tracks.

Fair enough.

 
 
 Also, as Eric (C) pointed out, one of the things which can change is which
 of several available versions of the content is being rendered (for 
 adaptive
 bitrate cases). This doesn't necessarily change any of the metadata
 currently exposed on the video element, but nevertheless it's information
 that the application may need. It would be nice to expose some kind of
 identifier for the currently rendered stream and have an event when this
 changes. I think that a stream-format-supplied identifier would be
 sufficient.
 
 I don't know about the adaptive streaming situation. I think that is
 more about statistics/metrics rather than about change of resource.
 All the alternatives in an adaptive streaming resource should
 provide the same number of tracks and the same video dimensions, just
 at different bitrate/quality, no?
 
 I think of the different adaptive versions on a per-track basis (i.e. the
 alternatives are *within* each track), not a bunch of alternatives each of
 which contains several tracks. Both are possible, of course.
 
 It's certainly possible (indeed common) for different bitrate video
 encodings to have different resolutions - there are video encoding reasons
 to do this. Of course the aspect ratio should not change and nor should the
 dimensions on the screen (both would be a little peculiar for the user).
 
 Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not
 the same as the resolution (for a start, they are in CSS pixels, which are
 square), but I think it quite likely that if the resolution of the video
 changes than the videoWidth and videoHeight might change. I'd be interested
 to hear how existing implementations relate resolution to videoWidth and
 videoHeight.
 
 Well, if videoWidth and videoHeight change and no dimensions on the
 video are provided through CSS, then surely the video will change size
 and the display will shrink. That would be a terrible user experience.
 For that reason I would suggest that such a change not be made in
 alternative adaptive streams.

That seems backwards to me! I would say For that reason I would suggest that 
dimensions are provided through CSS or through the width and height attributes.

Alternatively, we change the specification of the video element to accommodate 
this aspect of adaptive streaming (for example, the videoWidth and videoHeight 
could be defined to be based on the highest resolution bitrate being 
considered.)

There are good video encoding reasons for different bitrates to be encoded at 
different resolutions which are far more important than any reasons not to do 
either of the above.

 
 
 Different video dimensions should be
 provided through the source element and @media attribute, but within
 an adaptive stream, the alternatives should be consistent because the
 target device won't change. I guess this is a discussion for another
 thread... :-)
 
 Possibly ;-) The device knows much 

Re: [whatwg] Video feedback

[Silvia Pfeiffer]

On Tue, Jun 21, 2011 at 12:07 AM, Mark Watson wats...@netflix.com wrote:

 On Jun 20, 2011, at 11:52 AM, Silvia Pfeiffer wrote:

 On Mon, Jun 20, 2011 at 7:31 PM, Mark Watson wats...@netflix.com wrote:

 The TrackList object has an onchanged event, which I assumed would fire 
 when
 any of the information in the TrackList changes (e.g. tracks added or
 removed). But actually the spec doesn't state when this event fires (as 
 far
 as I could tell - unless it is implied by some general definition of 
 events
 called onchanged).

 Should there be some clarification here ?

 I understood that to relate to a change of cues only, since it is on
 the tracklist. I.e. it's an aggregate event from the oncuechange event
 of a cue inside the track. I didn't think it would relate to a change
 of existence of that track.

 Note that the even is attached to the TrackList, not the TrackList[],
 so it cannot be raised when a track is added or removed, only when
 something inside the TrackList changes.

 Are we talking about the same thing ? There is no TrackList array and
 TrackList is only used for audio/video, not text, so I don't understand the
 comment about cues.
 I'm talking
 about 
 http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist
  which
 is the base class for MultipleTrackList and ExclusiveTrackList used to
 represent all the audio and video tracks (respectively). One instance of the
 object represents all the tracks, so I would assume that a change in the
 number of tracks is a change to this object.

 Ah yes, you're right: I got confused.

 It says Whenever the selected track is changed, the user agent must
 queue a task to fire a simple event named change at the
 MultipleTrackList object. This means it fires when the selectedIndex
 is changed, i.e. the user chooses a different track for rendering. I
 still don't think it relates to changes in the composition of tracks
 of a resource. That should be something different and should probably
 be on the MediaElement and not on the track list to also cover changes
 in text tracks.

 Fair enough.



 Also, as Eric (C) pointed out, one of the things which can change is which
 of several available versions of the content is being rendered (for 
 adaptive
 bitrate cases). This doesn't necessarily change any of the metadata
 currently exposed on the video element, but nevertheless it's information
 that the application may need. It would be nice to expose some kind of
 identifier for the currently rendered stream and have an event when this
 changes. I think that a stream-format-supplied identifier would be
 sufficient.

 I don't know about the adaptive streaming situation. I think that is
 more about statistics/metrics rather than about change of resource.
 All the alternatives in an adaptive streaming resource should
 provide the same number of tracks and the same video dimensions, just
 at different bitrate/quality, no?

 I think of the different adaptive versions on a per-track basis (i.e. the
 alternatives are *within* each track), not a bunch of alternatives each of
 which contains several tracks. Both are possible, of course.

 It's certainly possible (indeed common) for different bitrate video
 encodings to have different resolutions - there are video encoding reasons
 to do this. Of course the aspect ratio should not change and nor should the
 dimensions on the screen (both would be a little peculiar for the user).

 Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not
 the same as the resolution (for a start, they are in CSS pixels, which are
 square), but I think it quite likely that if the resolution of the video
 changes than the videoWidth and videoHeight might change. I'd be interested
 to hear how existing implementations relate resolution to videoWidth and
 videoHeight.

 Well, if videoWidth and videoHeight change and no dimensions on the
 video are provided through CSS, then surely the video will change size
 and the display will shrink. That would be a terrible user experience.
 For that reason I would suggest that such a change not be made in
 alternative adaptive streams.

 That seems backwards to me! I would say For that reason I would suggest that 
 dimensions are provided through CSS or through the width and height 
 attributes.

 Alternatively, we change the specification of the video element to 
 accommodate this aspect of adaptive streaming (for example, the videoWidth 
 and videoHeight could be defined to be based on the highest resolution 
 bitrate being considered.)

 There are good video encoding reasons for different bitrates to be encoded at 
 different resolutions which are far more important than any reasons not to do 
 either of the above.



 Different video dimensions should be
 provided through the source element and @media attribute, but within
 an adaptive stream, the alternatives should be consistent because the
 target device won't change. I guess this is a 

Re: [whatwg] Video feedback

[Mark Watson]

On Jun 20, 2011, at 5:28 PM, Silvia Pfeiffer wrote:

 On Tue, Jun 21, 2011 at 12:07 AM, Mark Watson wats...@netflix.com wrote:
 
 On Jun 20, 2011, at 11:52 AM, Silvia Pfeiffer wrote:
 
 On Mon, Jun 20, 2011 at 7:31 PM, Mark Watson wats...@netflix.com wrote:
 
 The TrackList object has an onchanged event, which I assumed would fire 
 when
 any of the information in the TrackList changes (e.g. tracks added or
 removed). But actually the spec doesn't state when this event fires (as 
 far
 as I could tell - unless it is implied by some general definition of 
 events
 called onchanged).
 
 Should there be some clarification here ?
 
 I understood that to relate to a change of cues only, since it is on
 the tracklist. I.e. it's an aggregate event from the oncuechange event
 of a cue inside the track. I didn't think it would relate to a change
 of existence of that track.
 
 Note that the even is attached to the TrackList, not the TrackList[],
 so it cannot be raised when a track is added or removed, only when
 something inside the TrackList changes.
 
 Are we talking about the same thing ? There is no TrackList array and
 TrackList is only used for audio/video, not text, so I don't understand the
 comment about cues.
 I'm talking
 about 
 http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist
  which
 is the base class for MultipleTrackList and ExclusiveTrackList used to
 represent all the audio and video tracks (respectively). One instance of 
 the
 object represents all the tracks, so I would assume that a change in the
 number of tracks is a change to this object.
 
 Ah yes, you're right: I got confused.
 
 It says Whenever the selected track is changed, the user agent must
 queue a task to fire a simple event named change at the
 MultipleTrackList object. This means it fires when the selectedIndex
 is changed, i.e. the user chooses a different track for rendering. I
 still don't think it relates to changes in the composition of tracks
 of a resource. That should be something different and should probably
 be on the MediaElement and not on the track list to also cover changes
 in text tracks.
 
 Fair enough.
 
 
 
 Also, as Eric (C) pointed out, one of the things which can change is 
 which
 of several available versions of the content is being rendered (for 
 adaptive
 bitrate cases). This doesn't necessarily change any of the metadata
 currently exposed on the video element, but nevertheless it's information
 that the application may need. It would be nice to expose some kind of
 identifier for the currently rendered stream and have an event when this
 changes. I think that a stream-format-supplied identifier would be
 sufficient.
 
 I don't know about the adaptive streaming situation. I think that is
 more about statistics/metrics rather than about change of resource.
 All the alternatives in an adaptive streaming resource should
 provide the same number of tracks and the same video dimensions, just
 at different bitrate/quality, no?
 
 I think of the different adaptive versions on a per-track basis (i.e. the
 alternatives are *within* each track), not a bunch of alternatives each of
 which contains several tracks. Both are possible, of course.
 
 It's certainly possible (indeed common) for different bitrate video
 encodings to have different resolutions - there are video encoding reasons
 to do this. Of course the aspect ratio should not change and nor should the
 dimensions on the screen (both would be a little peculiar for the user).
 
 Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not
 the same as the resolution (for a start, they are in CSS pixels, which are
 square), but I think it quite likely that if the resolution of the video
 changes than the videoWidth and videoHeight might change. I'd be interested
 to hear how existing implementations relate resolution to videoWidth and
 videoHeight.
 
 Well, if videoWidth and videoHeight change and no dimensions on the
 video are provided through CSS, then surely the video will change size
 and the display will shrink. That would be a terrible user experience.
 For that reason I would suggest that such a change not be made in
 alternative adaptive streams.
 
 That seems backwards to me! I would say For that reason I would suggest 
 that dimensions are provided through CSS or through the width and height 
 attributes.
 
 Alternatively, we change the specification of the video element to 
 accommodate this aspect of adaptive streaming (for example, the videoWidth 
 and videoHeight could be defined to be based on the highest resolution 
 bitrate being considered.)
 
 There are good video encoding reasons for different bitrates to be encoded 
 at different resolutions which are far more important than any reasons not 
 to do either of the above.
 
 
 
 Different video dimensions should be
 provided through the source element and @media attribute, but within
 an adaptive stream, the alternatives 

Re: [whatwg] Video feedback

[Simon Pieters]

On Thu, 09 Jun 2011 03:47:49 +0200, Silvia Pfeiffer  
silviapfeiff...@gmail.com wrote:


For commercial video providers, the tracks in a live stream change all  
the time; this is not limited to audio and video tracks but would  
include text tracks as well.


OK, all this indicates to me that we probably want a metadatachanged
event to indicate there has been a change and that JS may need to
check some of its assumptions.


We already have durationchange. Duration is metadata. If we want to  
support changes to width/height, and the script is interested in when that  
happens, maybe there should be a dimensionchange event (but what's the use  
case for changing width/height mid-stream?). Does the spec support changes  
to text tracks mid-stream?


--
Simon Pieters
Opera Software