Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Sep 28, 2016 at 5:18 PM, Garrett Smith  wrote:
> On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
>> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>>
>>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>>> time-stretching (which is speed change with pitch compensation).
>>
>> That's neat. If you're interested in exposing this as an API on
>> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> might work?
>>
>
> Please post a link to that spec bug.

I don't think a bug was ever filed, but you can file one at
https://github.com/whatwg/html/issues

-- 
Philip Jägenstedt

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>
>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>> time-stretching (which is speed change with pitch compensation).
>
> That's neat. If you're interested in exposing this as an API on
> HTMLMediaElement, can you file a spec bug with a proposal of how it
> might work?
>

Please post a link to that spec bug.

Thanks,
-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

All,

There is no way to independently adjust pitch and speed of HTML5 video.
(Now, Chrome doesn't support preservesPitch (in any prefix), so things are
worse.)

Generally it is a good idea to read all of the thread's messages before
replying.

Garrett: Which implementations does this work in?
Jer: None, because there is no such thing as a PitchShiftNode.

Further discussion of the proposed API followed with comments of Jer's
proposed API being a lot work to do something simple.

Domenic should have read that, just like he should have read (and heeded)
the warning in my response to his harassing personal emails, to not email
me anymore, ever again.

Thank you,

On Wed, Jul 20, 2016 at 9:48 PM, Domenic Denicola  wrote:

> From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of
> Garrett Smith
>
> > What is the status on this?
>
> I believe the conclusion of the thread was that this is possible with the
> Web Audio API.
>
>


-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Garrett Smith

> What is the status on this?

I believe the conclusion of the thread was that this is possible with the Web 
Audio API.

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

What is the status on this?

Need to be able to independently adjust pitch and slow down playback.
I need this on a daily basis.

In Chrome pitch is now preserved regardless of setting
vid.[prefix]preservesPitch = false.



On Mon, Mar 14, 2016 at 1:13 AM, Xabier Rodríguez Calvar
 wrote:
> Hi,
>
> O Ven, 11-03-2016 ás 13:19 -0800, Jer Noble escribiu:
>> > Oh. Poor hardware integration, and now this…. Being Apple CEO is
>> not Tim Cook's greatest gift…
>>
>> I’m not sure how that’s relevant.
>
> Yes I agree with Jer here, it looks like a very bad, undesired and
> irrelevant joke.
>
> Best regards.



-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 11, 2016 at 1:19 PM, Jer Noble  wrote:
>
> On Mar 11, 2016, at 1:11 PM, Garrett Smith  wrote:
>

…

> I also found:—
> https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVAudioUnitTimePitch_Class/
> ("preliminary document for an API or technology in development.”)
>
>
> You’ll be interested to know that the design of the Web Audio API was
> heavily influenced by the design of the OS X Core Audio library, and that is
> class you’ve found is an Objective-C wrapper around a Core Audio node.  So
> yes, this class could well be used to implement a Web Audio PitchShift node.
>
> -Jer

If a PitchShift Node can do the job, then great! But, the API should
be much more straightforward, such as with an Adapter, so we have
playbackRate and playbackSpeed, and not a long string of a.b.c().d …

On the VIDEO — separate sliders for pitch and for playback rate. One
slider for speed (playback rate) and another for pitch. The sliders
MUST NOT affect each others' behavior. Imagine you're using the video
player and the speed slider affects the pitch, but only after you've
used the pitch slider affect. Obviously no good. And that is why it
doesn't exist.


Thank you,
-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Xabier Rodríguez Calvar]

Hi,

O Ven, 11-03-2016 ás 13:19 -0800, Jer Noble escribiu:
> > Oh. Poor hardware integration, and now this…. Being Apple CEO is
> not Tim Cook's greatest gift… 
> 
> I’m not sure how that’s relevant.

Yes I agree with Jer here, it looks like a very bad, undesired and
irrelevant joke.

Best regards.

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 11, 2016, at 1:11 PM, Garrett Smith  wrote:
> 
> 
> 
> On Tuesday, March 8, 2016, Jer Noble  > wrote:
> 
> > On Mar 8, 2016, at 4:42 PM, Garrett Smith > wrote:
> >
> > On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble > wrote:
> >>
> >>> On Mar 4, 2016, at 3:19 PM, Garrett Smith > 
> >>> wrote:
> >>>
> >>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble > wrote:
> 
> > On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt > 
> > wrote:
> >
> > On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  > <>> wrote:
> >> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  >> <>> wrote:
> >>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> >>> >
> >>> wrote:
>  On 11/12/15, Philip Jägenstedt > wrote:
> > On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> > >
> > wrote:
> >>
> >> On 10/19/15, Philip Jägenstedt > wrote:
> >>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> >>> >
> >>> wrote:
>  On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>  >
>  wrote:
> > From: Eric Carlson [mailto:eric.carl...@apple.com <>]
> 
>  
> >>
> >> The Web Audio equivalent would be:
> >>
> >> var video = document.querySelector(‘video’);
> >> video.preservesPitch = false;
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> >
> > Which implementations does that work in?
> 
> None, because there is no such thing as a PitchShiftNode.
> 
> 
> I see. 
>  
> > That code is more complex than should be necessary. I see where you're
> > coming from on separating the audio. Could we move the media decorator
> > behind the scenes, and replace it with a simple getter/setter property
> > like `videoElement.audio` so that that can happen automagically?
> > Reminds me of createElement, createRange, document.implementation,
> > etc. Warts!
> 
> I’m not entirely sure what you’re asking here. If it’s that you don’t like 
> the `context.createMediaElementSource()` or `context.createPitchShift()` 
> syntax and would rather a constructor syntax, Issue #250 
>  > in the Web Audio spec 
> is the issue for you.
> 
> > But then again, you also just said that there are no APIs on OS X that
> > allow an arbitrary pitch shift to be added to audio. If that is true,
> > then your `createPitchShift` code would be possible anyway, is that
> 
> There is no such API for such post-processing built into the OS X media 
> frameworks. 
> 
> Oh. Poor hardware integration, and now this…. Being Apple CEO is not Tim 
> Cook's greatest gift… 

I’m not sure how that’s relevant.

> As an example, there is an API for preserving pitch across rate changes: 
> -[AVPlayerItem setAudioTimePitchAlgorithm:] 
>   
> >.
>   So implementing the HTMLMediaElement “preservesPitch" property is trivial.  
> There is no such API for applying a specific and arbitrary amount of pitch 
> shifting. It would have to be implemented entirely by hand, the same way we 
> implement processing of media element audio with the Web Audio API.  Which 
> gets back to: at this point, we should just implement a Pitch Shift node in 
> Web Audio.
> 
> 
>  I see.
> 
> >> var video = document.querySelector(‘video’);
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> 
> Too many steps and implementation details. It's complicated. It's asking for 
> more js libraries and wrappers for that, and other situations related to 
> videos' audio.  Make it simpler. Hide the implementation details.
> 
> Hide:—
>  * AudioContext creation and subsequent video connection.
>  * createPitchShift — use audio.pitchShift to get a pitchShiftNode
>  * pitchShiftNode.shiftAmount — 

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Tuesday, March 8, 2016, Jer Noble  wrote:

>
> > On Mar 8, 2016, at 4:42 PM, Garrett Smith 
> wrote:
> >
> > On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble  wrote:
> >>
> >>> On Mar 4, 2016, at 3:19 PM, Garrett Smith 
> wrote:
> >>>
> >>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
> 
> > On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt 
> wrote:
> >
> > On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith <
> dhtmlkitc...@gmail.com> wrote:
> >> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt <
> phil...@opera.com> wrote:
> >>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
> >>> wrote:
>  On 11/12/15, Philip Jägenstedt  wrote:
> > On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
> > wrote:
> >>
> >> On 10/19/15, Philip Jägenstedt  wrote:
> >>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <
> phil...@opera.com>
> >>> wrote:
>  On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola <
> d...@domenic.me>
>  wrote:
> > From: Eric Carlson [mailto:eric.carl...@apple.com]


>
>
>>
> >> The Web Audio equivalent would be:
> >>
> >> var video = document.querySelector(‘video’);
> >> video.preservesPitch = false;
> >> var context = new AudioContext();
> >> var sourceNode = context.createMediaElementSource(video);
> >> var pitchShiftNode = context.createPitchShift();
> >> pitchShiftNode.shiftAmount = 1.023;
> >> sourceNode.connect(pitchShiftNode);
> >> pitchShiftNode.connect(context.destination);
> >>
> >
> > Which implementations does that work in?
>
> None, because there is no such thing as a PitchShiftNode.
>
>
I see.


> > That code is more complex than should be necessary. I see where you're
> > coming from on separating the audio. Could we move the media decorator
> > behind the scenes, and replace it with a simple getter/setter property
> > like `videoElement.audio` so that that can happen automagically?
> > Reminds me of createElement, createRange, document.implementation,
> > etc. Warts!
>
> I’m not entirely sure what you’re asking here. If it’s that you don’t like
> the `context.createMediaElementSource()` or `context.createPitchShift()`
> syntax and would rather a constructor syntax, Issue #250 <
> https://github.com/WebAudio/web-audio-api/issues/250> in the Web Audio
> spec is the issue for you.
>
> > But then again, you also just said that there are no APIs on OS X that
> > allow an arbitrary pitch shift to be added to audio. If that is true,
> > then your `createPitchShift` code would be possible anyway, is that
>
> There is no such API for such post-processing built into the OS X media
> frameworks.


Oh. Poor hardware integration, and now this…. Being Apple CEO is not Tim
Cook's greatest gift…

As an example, there is an API for preserving pitch across rate changes:
> -[AVPlayerItem setAudioTimePitchAlgorithm:] <
> https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVPlayerItem_Class/index.html#//apple_ref/occ/instp/AVPlayerItem/audioTimePitchAlgorithm>.
> So implementing the HTMLMediaElement “preservesPitch" property is trivial.
> There is no such API for applying a specific and arbitrary amount of pitch
> shifting. It would have to be implemented entirely by hand, the same way we
> implement processing of media element audio with the Web Audio API.  Which
> gets back to: at this point, we should just implement a Pitch Shift node in
> Web Audio.
>
>
 I see.

>> var video = document.querySelector(‘video’);
>> var context = new AudioContext();
>> var sourceNode = context.createMediaElementSource(video);
>> var pitchShiftNode = context.createPitchShift();
>> pitchShiftNode.shiftAmount = 1.023;
>> sourceNode.connect(pitchShiftNode);
>> pitchShiftNode.connect(context.destination);
>>

Too many steps and implementation details. It's complicated. It's asking
for more js libraries and wrappers for that, and other situations related
to videos' audio.  Make it simpler. Hide the implementation details.

Hide:—
 * AudioContext creation and subsequent video connection.
 * createPitchShift — use audio.pitchShift to get a pitchShiftNode
 * pitchShiftNode.shiftAmount — make pitchShiftNode a getter/setter (like
video.playbackRate)

>> var video = document.querySelector(‘video’);
>> var context = new AudioContext();
>> var sourceNode = context.createMediaElementSource(video);
>> var context = new AudioContext();

// Hidden AudioContext construction
var audio = video.audio;

>> var pitchShiftNode = context.createPitchShift();
>> pitchShiftNode.shiftAmount = 1.023;

//  Hidden AudioContext createPitchShift factory.
//  Setter on pitchShift
audio.pitchShift = 1.023;

>> sourceNode.connect(pitchShiftNode);
>> 

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble  wrote:
>
>> On Mar 4, 2016, at 3:19 PM, Garrett Smith  wrote:
>>
>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>>>
 On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:

 On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  
 wrote:
> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
> wrote:
>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>> wrote:
>>> On 11/12/15, Philip Jägenstedt  wrote:
 On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
 wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>> 
>> wrote:
>>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>>> wrote:
 From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>> Two things.
>>
>> 1. Do the underlying media frameworks that browsers are using support
>> arbitrary pitch changes, or do they also only have the limited
>> preservesPitch-style API?
>
> Are there any problems getting in the way of pitch adjustment (without
> depending on playbackRate)?

 I don't know, that was basically my question too. If the underlying
 APIs don't support it, that's a problem that needs to be fixed first.
>>>
>>>
>>> There are no such APIs on OS X which would allow an arbitrary pitch shift 
>>> to be added to an otherwise normally playing piece of audio.
>>>
>>> IMO, this is a more appropriate request for the Web Audio API (adding a 
>>> Audio Node which can add an arbitrary amount of pitch shift).  At which 
>>> point, there would be no need for this in HTMLMediaElement, as authors 
>>> could make a simple node graph consisting of an MediaElementAudioSourceNode 
>>> and a PitchShiftNode.
>>>
>>> -Jer
>>
>> But that can't work on OSX, right?  I wonder how audio software on mac
>> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
>> many others can all do this.
>
> None of them use the built in platform APIs to shift the pitch of encoded 
> media.  Each does it manually, within their app, and each probably uses a 
> different algorithm to achieve shift in pitch.
>
>> Plus how would web Audio API solve for the use case?
>>
>> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
>> song is about a 50 cents flat of standard tuning. The pitch can be
>> adjusted by setting playbackRate to 1.023 and setting
>> MozPreservesPitch to false:—
>>
>> var vv = document.querySelector("video");
>> vv.mozPreservesPitch = 0;
>> vv.playbackRate = 1.023
>>
>> — but that speeds it up. I don't want speed coupled with pitch.
>
> The Web Audio equivalent would be:
>
> var video = document.querySelector(‘video’);
> video.preservesPitch = false;
> var context = new AudioContext();
> var sourceNode = context.createMediaElementSource(video);
> var pitchShiftNode = context.createPitchShift();
> pitchShiftNode.shiftAmount = 1.023;
> sourceNode.connect(pitchShiftNode);
> pitchShiftNode.connect(context.destination);
>

Which implementations does that work in?

That code is more complex than should be necessary. I see where you're
coming from on separating the audio. Could we move the media decorator
behind the scenes, and replace it with a simple getter/setter property
like `videoElement.audio` so that that can happen automagically?
Reminds me of createElement, createRange, document.implementation,
etc. Warts!

But then again, you also just said that there are no APIs on OS X that
allow an arbitrary pitch shift to be added to audio. If that is true,
then your `createPitchShift` code would be possible anyway, is that
right?

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 4, 2016, at 3:19 PM, Garrett Smith  wrote:
> 
> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>> 
>>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
>>> 
>>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  
>>> wrote:
 On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
 wrote:
> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> wrote:
>> On 11/12/15, Philip Jägenstedt  wrote:
>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>>> wrote:
 
 On 10/19/15, Philip Jägenstedt  wrote:
> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> wrote:
>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>> wrote:
>>> From: Eric Carlson [mailto:eric.carl...@apple.com]
 
> Two things.
> 
> 1. Do the underlying media frameworks that browsers are using support
> arbitrary pitch changes, or do they also only have the limited
> preservesPitch-style API?
 
 Are there any problems getting in the way of pitch adjustment (without
 depending on playbackRate)?
>>> 
>>> I don't know, that was basically my question too. If the underlying
>>> APIs don't support it, that's a problem that needs to be fixed first.
>> 
>> 
>> There are no such APIs on OS X which would allow an arbitrary pitch shift to 
>> be added to an otherwise normally playing piece of audio.
>> 
>> IMO, this is a more appropriate request for the Web Audio API (adding a 
>> Audio Node which can add an arbitrary amount of pitch shift).  At which 
>> point, there would be no need for this in HTMLMediaElement, as authors could 
>> make a simple node graph consisting of an MediaElementAudioSourceNode and a 
>> PitchShiftNode.
>> 
>> -Jer
> 
> But that can't work on OSX, right?  I wonder how audio software on mac
> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
> many others can all do this.

None of them use the built in platform APIs to shift the pitch of encoded 
media.  Each does it manually, within their app, and each probably uses a 
different algorithm to achieve shift in pitch.

> Plus how would web Audio API solve for the use case?
> 
> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
> song is about a 50 cents flat of standard tuning. The pitch can be
> adjusted by setting playbackRate to 1.023 and setting
> MozPreservesPitch to false:—
> 
> var vv = document.querySelector("video");
> vv.mozPreservesPitch = 0;
> vv.playbackRate = 1.023
> 
> — but that speeds it up. I don't want speed coupled with pitch.

The Web Audio equivalent would be:

var video = document.querySelector(‘video’);
video.preservesPitch = false;
var context = new AudioContext();
var sourceNode = context.createMediaElementSource(video);
var pitchShiftNode = context.createPitchShift();
pitchShiftNode.shiftAmount = 1.023;
sourceNode.connect(pitchShiftNode);
pitchShiftNode.connect(context.destination);

-Jer

> Thanks,
> -- 
> Garrett
> @xkit
> ChordCycles.wordpress.com
> garretts.github.io
> personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble  wrote:
>
>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
>>
>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
>>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
>>> wrote:
 On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
 wrote:
> On 11/12/15, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>> wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
 On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>>
 Two things.

 1. Do the underlying media frameworks that browsers are using support
 arbitrary pitch changes, or do they also only have the limited
 preservesPitch-style API?
>>>
>>> Are there any problems getting in the way of pitch adjustment (without
>>> depending on playbackRate)?
>>
>> I don't know, that was basically my question too. If the underlying
>> APIs don't support it, that's a problem that needs to be fixed first.
>
>
> There are no such APIs on OS X which would allow an arbitrary pitch shift to 
> be added to an otherwise normally playing piece of audio.
>
> IMO, this is a more appropriate request for the Web Audio API (adding a Audio 
> Node which can add an arbitrary amount of pitch shift).  At which point, 
> there would be no need for this in HTMLMediaElement, as authors could make a 
> simple node graph consisting of an MediaElementAudioSourceNode and a 
> PitchShiftNode.
>
> -Jer

But that can't work on OSX, right?  I wonder how audio software on mac
does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
many others can all do this.

Plus how would web Audio API solve for the use case?

To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
song is about a 50 cents flat of standard tuning. The pitch can be
adjusted by setting playbackRate to 1.023 and setting
MozPreservesPitch to false:—

var vv = document.querySelector("video");
vv.mozPreservesPitch = 0;
vv.playbackRate = 1.023

— but that speeds it up. I don't want speed coupled with pitch.

Thanks,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Jer Noble]

> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt  wrote:
> 
> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  
>> wrote:
>>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>>> wrote:
 On 11/12/15, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> wrote:
>> 
>> On 10/19/15, Philip Jägenstedt  wrote:
>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> wrote:
 On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
 wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
>> 
>>> Two things.
>>> 
>>> 1. Do the underlying media frameworks that browsers are using support
>>> arbitrary pitch changes, or do they also only have the limited
>>> preservesPitch-style API?
>> 
>> Are there any problems getting in the way of pitch adjustment (without
>> depending on playbackRate)?
> 
> I don't know, that was basically my question too. If the underlying
> APIs don't support it, that's a problem that needs to be fixed first.


There are no such APIs on OS X which would allow an arbitrary pitch shift to be 
added to an otherwise normally playing piece of audio.

IMO, this is a more appropriate request for the Web Audio API (adding a Audio 
Node which can add an arbitrary amount of pitch shift).  At which point, there 
would be no need for this in HTMLMediaElement, as authors could make a simple 
node graph consisting of an MediaElementAudioSourceNode and a PitchShiftNode.

-Jer

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Thu, Mar 3, 2016 at 1:18 AM, Paul Adenot  wrote:
> That looks appropriate, and is in line with how the Web Audio API does
> detuning (using the same unit, [0]).
>
> That said, I'm not sure about the use case here? Time-stretching (normal
> playbackRate) is useful, for example, to watch a conference slightly faster
> in order to save some time. I can't really find non-musical use-cases, and
> musical use-cases are maybe better served using the Web Audio API.
>

In summary:

User wants to hear the audio at the desired pitch, for accompaniment
or learning purposes, ideally while simultaneously watching the video
at the desired speed, using a video sharing or lesson websites.

User finds a version of the artist performing the piece among other
versions and lessons in various tunings. The user uses adjusts video's
pitch, independently of playback speed, to match his instrument's
tuning using.

The pitch control can be provided by a user script, bookmarklet, or extension.*

Existing software for this includes Amazing Slow Downer.

(more below)

Thank you,


> Paul.
>
> [0]:
> http://webaudio.github.io/web-audio-api/#widl-AudioBufferSourceNode-detune
>
> On Thu, Mar 3, 2016 at 12:00 AM, Garrett Smith 
> wrote:
>>
>> On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith 
>> wrote:
>> > On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt 
>> > wrote:
>> >> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot 
>> >> wrote:
>> >>>
>> >>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to
>> >>> time-stretching (which is speed change with pitch compensation).
>> >>
>> >> That's neat. If you're interested in exposing this as an API on
>> >> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> >> might work?
>> >>
>> >
>> > Here is what I had in mind; hope it helps.
>> >
>> > Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
>> > pitch in cents.
>> >
>> > Equally tempered semitones span 100 cents.
>> >
>> > The developer can use INPUT type="range" with an oninput, to allow the
>> > user to adjust the video's pitch, in cents.
>> >
>> > interface HTMLMediaElement : HTMLElement {
>> >  attribute unsigned short pitchAdjustment;
>> >  …
>> > }
>> >
>>
>> Not unsigned, sorry.
>>
>> interface HTMLMediaElement : HTMLElement {
>>   attribute short pitchAdjustment;
>>   …
>>  }
>>
>> That should allow for pitch adjustment over 54 octaves. This is
>> withing human hearing range of 10 octaves. 1 having the value of one
>> cent, gives 1200 cents per octave.
>>
>> let
>>  semitone = 100, // cents
>>  octave = 12 * semitone, // 1200
>>  sizeOfShort = 65535;
>> sizeOfShort / octave 54.6125
>>
>> Signed short range is −32768 to 32767
>> --
>> Garrett
>> @xkit
>> ChordCycles.wordpress.com
>> garretts.github.io
>> personx.tumblr.com
>
>



-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Paul Adenot]

That looks appropriate, and is in line with how the Web Audio API does
detuning (using the same unit, [0]).

That said, I'm not sure about the use case here? Time-stretching (normal
playbackRate) is useful, for example, to watch a conference slightly faster
in order to save some time. I can't really find non-musical use-cases, and
musical use-cases are maybe better served using the Web Audio API.

Paul.

[0]:
http://webaudio.github.io/web-audio-api/#widl-AudioBufferSourceNode-detune

On Thu, Mar 3, 2016 at 12:00 AM, Garrett Smith 
wrote:

> On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith 
> wrote:
> > On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt 
> wrote:
> >> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot 
> wrote:
> >>>
> >>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to
> time-stretching (which is speed change with pitch compensation).
> >>
> >> That's neat. If you're interested in exposing this as an API on
> >> HTMLMediaElement, can you file a spec bug with a proposal of how it
> >> might work?
> >>
> >
> > Here is what I had in mind; hope it helps.
> >
> > Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
> > pitch in cents.
> >
> > Equally tempered semitones span 100 cents.
> >
> > The developer can use INPUT type="range" with an oninput, to allow the
> > user to adjust the video's pitch, in cents.
> >
> > interface HTMLMediaElement : HTMLElement {
> >  attribute unsigned short pitchAdjustment;
> >  …
> > }
> >
>
> Not unsigned, sorry.
>
> interface HTMLMediaElement : HTMLElement {
>   attribute short pitchAdjustment;
>   …
>  }
>
> That should allow for pitch adjustment over 54 octaves. This is
> withing human hearing range of 10 octaves. 1 having the value of one
> cent, gives 1200 cents per octave.
>
> let
>  semitone = 100, // cents
>  octave = 12 * semitone, // 1200
>  sizeOfShort = 65535;
> sizeOfShort / octave 54.6125
>
> Signed short range is −32768 to 32767
> --
> Garrett
> @xkit
> ChordCycles.wordpress.com
> garretts.github.io
> personx.tumblr.com
>

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 1:09 PM, Garrett Smith  wrote:
> On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
>> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>>
>>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>>> time-stretching (which is speed change with pitch compensation).
>>
>> That's neat. If you're interested in exposing this as an API on
>> HTMLMediaElement, can you file a spec bug with a proposal of how it
>> might work?
>>
>
> Here is what I had in mind; hope it helps.
>
> Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
> pitch in cents.
>
> Equally tempered semitones span 100 cents.
>
> The developer can use INPUT type="range" with an oninput, to allow the
> user to adjust the video's pitch, in cents.
>
> interface HTMLMediaElement : HTMLElement {
>  attribute unsigned short pitchAdjustment;
>  …
> }
>

Not unsigned, sorry.

interface HTMLMediaElement : HTMLElement {
  attribute short pitchAdjustment;
  …
 }

That should allow for pitch adjustment over 54 octaves. This is
withing human hearing range of 10 octaves. 1 having the value of one
cent, gives 1200 cents per octave.

let
 semitone = 100, // cents
 octave = 12 * semitone, // 1200
 sizeOfShort = 65535;
sizeOfShort / octave 54.6125

Signed short range is −32768 to 32767
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Wed, Mar 2, 2016 at 2:36 AM, Philip Jägenstedt  wrote:
> On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>>
>> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
>> time-stretching (which is speed change with pitch compensation).
>
> That's neat. If you're interested in exposing this as an API on
> HTMLMediaElement, can you file a spec bug with a proposal of how it
> might work?
>

Here is what I had in mind; hope it helps.

Create a new property `pitchAdjustment` on HTMLMediaElement to adjust
pitch in cents.

Equally tempered semitones span 100 cents.

The developer can use INPUT type="range" with an oninput, to allow the
user to adjust the video's pitch, in cents.

interface HTMLMediaElement : HTMLElement {
 attribute unsigned short pitchAdjustment;
 …
}

pitchAdjustment adjusts the target HTMLMediaElement's audio pitch
relative to its original pitch, in cents. A value of 0 represents no
pitch adjustment.

Getting gets the last set value; 0 if unset. Setting sets the value
and changes the pitch in cents (relative to the media element's
original pitch).

Example:

http://example.net/; id="vv"">




function updateMoviePitch(ev) {
  var vv = document.getElementById("vv");
  vv.pitchAdjustment = +this.value;
  showPitchUI(vv, this.value);
}

Thank you,
-- 
Garrett
@xkit

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Mar 2, 2016 at 5:08 PM, Paul Adenot  wrote:
>
> Gecko uses code that can do arbitrary pitch adjustment, unrelated to 
> time-stretching (which is speed change with pitch compensation).

That's neat. If you're interested in exposing this as an API on
HTMLMediaElement, can you file a spec bug with a proposal of how it
might work?

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Paul Adenot]

Gecko uses code that can do arbitrary pitch adjustment, unrelated to
time-stretching (which is speed change with pitch compensation).

Paul.

On Wed, Mar 2, 2016 at 5:00 AM, Philip Jägenstedt  wrote:

> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith 
> wrote:
> > On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt 
> wrote:
> >> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith  >
> >> wrote:
> >>> On 11/12/15, Philip Jägenstedt  wrote:
>  On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <
> dhtmlkitc...@gmail.com>
>  wrote:
> >
> > On 10/19/15, Philip Jägenstedt  wrote:
> > > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <
> phil...@opera.com>
> > > wrote:
> > >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> > >> wrote:
> > >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
> >
> >> Two things.
> >>
> >> 1. Do the underlying media frameworks that browsers are using support
> >> arbitrary pitch changes, or do they also only have the limited
> >> preservesPitch-style API?
> >
> > Are there any problems getting in the way of pitch adjustment (without
> > depending on playbackRate)?
>
> I don't know, that was basically my question too. If the underlying
> APIs don't support it, that's a problem that needs to be fixed first.
>
> Philip
>

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith  wrote:
> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
>> wrote:
>>> On 11/12/15, Philip Jägenstedt  wrote:
 On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
 wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> > wrote:
> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
> >> wrote:
> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>> Two things.
>>
>> 1. Do the underlying media frameworks that browsers are using support
>> arbitrary pitch changes, or do they also only have the limited
>> preservesPitch-style API?
>
> Are there any problems getting in the way of pitch adjustment (without
> depending on playbackRate)?

I don't know, that was basically my question too. If the underlying
APIs don't support it, that's a problem that needs to be fixed first.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
> wrote:
>> On 11/12/15, Philip Jägenstedt  wrote:
>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>>> wrote:

 On 10/19/15, Philip Jägenstedt  wrote:
 > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 > wrote:
 >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
 >> wrote:
 >>> From: Eric Carlson [mailto:eric.carl...@apple.com]

> Two things.
>
> 1. Do the underlying media frameworks that browsers are using support
> arbitrary pitch changes, or do they also only have the limited
> preservesPitch-style API?

Are there any problems getting in the way of pitch adjustment (without
depending on playbackRate)?

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:43 AM, Eric Carlson  wrote:
>
>> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
>>
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith > > wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
 On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>
>>>
>>> [...]
 I've filed a spec issue to make it so:
 https://github.com/whatwg/html/issues/262

 If there's any implementor interest in pitch control that goes beyond
 (independently) or that, please file a separate issue.

>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>   I am actually quite ambivalent about this feature - it is currently broken 
> on OSX and it has never been implemented on iOS, but as far as I can see we 
> haven’t received a single bug report about this.

Ah. I removed webkitPreservesPitched entirely from Blink because it
wasn't implemented on any platforms, maybe you could do the same on
iOS?

More importantly, is it possible to support this on iOS, and is it
likely to bubble to the top of your priorities any time soon if it
were added to the spec? I don't think it's high priority for anyone
working on Chromium, but unless Safari and Firefox are ready to drop
their prefixed APIs entirely it still seems like a good idea to get
what little interop we can by standardizing.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
wrote:
> On 11/12/15, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>> wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
>>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> > wrote:
>>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>>> >> wrote:
>>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>> >>>
>>>
>>> [...]
>>> > I've filed a spec issue to make it so:
>>> > https://github.com/whatwg/html/issues/262
>>> >
>>> > If there's any implementor interest in pitch control that goes beyond
>>> > (independently) or that, please file a separate issue.
>>> >
>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>
> https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

I don't understand, isn't the AudioTrack API (enabling/disabling individual
audio tracks) separate a separate concern? Or are you saying that there's
no way to get the audio tracks separately into the Web Audio API in order
to process them differently? That is certainly true, and if having that
capability would allow you to solve the problem with the Web Audio API, I
think that seems like a powerful primitive to have.

>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>
> The situation where you want hear the audio at the desired pitch,
> while simultaneously watching the video at the desired speed, to see
> it performed, and to play along. It doesn't work well when trying to
> play to a video example, such as playing an open G chord, and the
> sound from the video does not match the sound from your own guitar.
>
> What I can do now on websites that use VIDEO, such as YouTube, is
> limited to using the web inspector (console). I click "inspect
> element" to find the VIDEO element, then change its DOM properties for
> playbackRate = .95 and mozPreservesPitch = false. That'll get
> everything down a half-step (although at 95% speed).
>
> For example, watch Andy James play "Harvester of Sorrow" but hear it
> 1/2 step down:
>
> https://www.youtube.com/watch?v=vYZjC_U1VJg
>
> 1) Right click on the video "Inspect Element"
> 2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
> 3) set playbackRate: 0.95, mozPreservesPitch: 0
>
> Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
> few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
> down further.
>
> A=440 (standard tuning) is much more common. When you want to learn
> something played tuned down 1/2 step and your guitar is in standard
> tuning, it is a problem. Constantly retuning the instrument is not
> practical. Especially for guitars with locking nut systems or for
> other instruments like piano. I don't know about learning horn parts.
> Or you can buy multiple instruments and maintain different tunings for
> them.
>
> Fast passages, it's helpful to slow the speed down but keep the pitch
> as original and in some cases, you want to slow the speed down and
> adjust the pitch separately, so you can do things like learning Yngwie
> Malmsteen songs while using a guitar in standard tuning.
>
> I realize there's not much demand for it. But there seems to be
> separation of playback pitch with `preservesPitch` and 

Re: [whatwg] VIDEO and pitchAdjustment

[Eric Carlson]

> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
> 
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  > wrote:
>> 
>> On 10/19/15, Philip Jägenstedt  wrote:
>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> wrote:
 On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
> 
>> 
>> [...]
>>> I've filed a spec issue to make it so:
>>> https://github.com/whatwg/html/issues/262
>>> 
>>> If there's any implementor interest in pitch control that goes beyond
>>> (independently) or that, please file a separate issue.
>>> 
>> 
>> They won't.
>> 
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>> 
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>> 
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>> 
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>> 
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>> 
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
> 
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
> 
> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
> 
  I am actually quite ambivalent about this feature - it is currently broken on 
OSX and it has never been implemented on iOS, but as far as I can see we 
haven’t received a single bug report about this.

eric

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 10/19/15, Philip Jägenstedt  wrote:
> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> wrote:
>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>>

[...]
> I've filed a spec issue to make it so:
> https://github.com/whatwg/html/issues/262
>
> If there's any implementor interest in pitch control that goes beyond
> (independently) or that, please file a separate issue.
>

They won't.

You can hold the standard of "they need to come here and post up
cogent arguments in favor of feature X", but it ain't gonna happen
that way.

There just isn't a whole lot of money in music education. How many
music education companies are W3C members?

Unlike technology companies like Facebook, Google, Nokia, Opera, and
other companies the post here, small music education operations like
artistworks, jammit, licklibrary, etc are more about their domain —
"music" — than they are about technology.

Major music education websites are still using Flash; their developers
are busy fixing broken links, making the login feature, database, etc
work, etc. Flash is not nice but they apparently were not funded or
motivated enough by the existing HTML5 HTMLMediaElement to use it
instead.

Control over playbackRate has a greater value than pitch control. But
those sites don't even allow the students to change the playbackRate
because they're still using Flash.

You won't read posts here about what students have to say about it the
value of having HTML5 vs Flash, or independent control over pitch and
playbackRate.

-- 
Garrett / independent voice
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> > wrote:
> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
> >>>
>
> [...]
> > I've filed a spec issue to make it so:
> > https://github.com/whatwg/html/issues/262
> >
> > If there's any implementor interest in pitch control that goes beyond
> > (independently) or that, please file a separate issue.
> >
>
> They won't.
>
> You can hold the standard of "they need to come here and post up
> cogent arguments in favor of feature X", but it ain't gonna happen
> that way.
>
> There just isn't a whole lot of money in music education. How many
> music education companies are W3C members?
>
> Unlike technology companies like Facebook, Google, Nokia, Opera, and
> other companies the post here, small music education operations like
> artistworks, jammit, licklibrary, etc are more about their domain —
> "music" — than they are about technology.
>
> Major music education websites are still using Flash; their developers
> are busy fixing broken links, making the login feature, database, etc
> work, etc. Flash is not nice but they apparently were not funded or
> motivated enough by the existing HTML5 HTMLMediaElement to use it
> instead.
>
> Control over playbackRate has a greater value than pitch control. But
> those sites don't even allow the students to change the playbackRate
> because they're still using Flash.
>
> You won't read posts here about what students have to say about it the
> value of having HTML5 vs Flash, or independent control over pitch and
> playbackRate.

Have you investigated whether you can achieve your use cases using the
Web Audio API? If it isn't possible, is there a small addition to Web
Audio that would solve the problem?

It is unfortunately quite hard to get all browser vendors (or even
one) interested in implementing support for something that is only
expected to benefit a niche use case, but we should strive to make
available the primitives that would it possible to implement yourself.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 11/12/15, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> wrote:
>>
>> On 10/19/15, Philip Jägenstedt  wrote:
>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>> > wrote:
>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>> >> wrote:
>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>> >>>
>>
>> [...]
>> > I've filed a spec issue to make it so:
>> > https://github.com/whatwg/html/issues/262
>> >
>> > If there's any implementor interest in pitch control that goes beyond
>> > (independently) or that, please file a separate issue.
>> >
>>
>> They won't.
>>
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>>
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>>
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>>
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>>
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>>
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
>
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
>

https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
>

The situation where you want hear the audio at the desired pitch,
while simultaneously watching the video at the desired speed, to see
it performed, and to play along. It doesn't work well when trying to
play to a video example, such as playing an open G chord, and the
sound from the video does not match the sound from your own guitar.

What I can do now on websites that use VIDEO, such as YouTube, is
limited to using the web inspector (console). I click "inspect
element" to find the VIDEO element, then change its DOM properties for
playbackRate = .95 and mozPreservesPitch = false. That'll get
everything down a half-step (although at 95% speed).

For example, watch Andy James play "Harvester of Sorrow" but hear it
1/2 step down:

https://www.youtube.com/watch?v=vYZjC_U1VJg

1) Right click on the video "Inspect Element"
2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
3) set playbackRate: 0.95, mozPreservesPitch: 0

Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
down further.

A=440 (standard tuning) is much more common. When you want to learn
something played tuned down 1/2 step and your guitar is in standard
tuning, it is a problem. Constantly retuning the instrument is not
practical. Especially for guitars with locking nut systems or for
other instruments like piano. I don't know about learning horn parts.
Or you can buy multiple instruments and maintain different tunings for
them.

Fast passages, it's helpful to slow the speed down but keep the pitch
as original and in some cases, you want to slow the speed down and
adjust the pitch separately, so you can do things like learning Yngwie
Malmsteen songs while using a guitar in standard tuning.

I realize there's not much demand for it. But there seems to be
separation of playback pitch with `preservesPitch` and `playbackRate`.
It would be much better with a control to adjustm pitch, if it is not
too much work.

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt  wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>
>>>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>>>
>>> ...
>>>
>>>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
>>> content.
>>
>> That's really helpful, thanks. Combined with Edge's keyframes-only support, 
>> it sounds like we should probably leave the spec as it is.
>>
>> Do you have thoughts on a mozPreservesPitch equivalent?
>
> Should we just standardize HTMLMediaElement.preservesPitch, perhaps?
> Note that WebKit also has webkitPreservesPitch, but I removed it from
> Blink because it didn't actually do anything in Chromium.
>
> In both Gecko and WebKit it defaults to true. Is there anything else
> worth knowing before writing the spec for this?

I've filed a spec issue to make it so:
https://github.com/whatwg/html/issues/262

If there's any implementor interest in pitch control that goes beyond
(independently) or that, please file a separate issue.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/27/15, Robert O'Callahan  wrote:
> On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith 
> wrote:
>

[...]

> But variable pitch control it would be useful for music adjustments
>> like "over the mountain",  "Black Star", "Take your Whiskey Home", all
>> originally recorded at half-step detuning (A=415), none of which match
>> "standard tuning" commonly used in schools, industry, etc. Recordings
>> of Baroque era music often use instruments tuned lower (and also
>> different scale temperment, but that is a different issue). So it
>> would be nice to adjust the pitch with a `pitchAdjustment` property,
>> as a double, to adjust pitch in cents.
>>
>
> I'd prefer to focus on making sure that Web Audio's Audio Workers are
> powerful enough for Web developers to implement this themselves.
>

I hear you, but I don't understand why.

To have control over audio pitch and playback rate, within a video,
with just a simple property, and without having to create a Worker,
would be very powerful for learning music.

Movie displays music notation or tablature, or a live performance. I
have used such.
The user can adjust the pitch, if necessary and change the speed, for learning.

We want to be able to adjust playbackRate and pitchAdjustment
independently. That might hypothetically be possible by directly
typing into the console, using user scripts, or bookmarklets. Online
music schools can build it into their HTML5 players (though today,
most of them still use the Flash player), for normal musicians (who
aren't using the console, etc).
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Michael Enright]

I'm an implementer (to the fullest extent that one can be one by
modifying existing open source), just following the convention of this
list of stating my use case. The use case reflects reality since it
was implemented last year.

On Mon, Aug 31, 2015 at 12:04 PM, Domenic Denicola  wrote:
> To be clear:
>
> Everyone can imagine use cases for playing videos backward. However, so far 
> the only statements we have about implementations are negative. My subthread 
> was more concerned with making the spec reflect current reality. If you can 
> convince implementers to support backward videos, then that's separate, and 
> we can change the spec again.
>

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> 
> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:
> 
>> QuickTime supports full variable speed playback and has done for well over
>> a decade. With bidirectionally predicted frames you need a fair few buffers
>> anyway, so generalising to full variable wait is easier than posters above
>> claim - you need to work a GOP at a time, but memory buffering isn't the
>> big issue these days.
>> 
> 
> "GOP”?

Group of Pictures.  Video-speak for the run between random access points.

> 
> How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
> keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> bytes = 4.32 GiB. Reading back those frames would kill performance so that
> all has to stay in VRAM. I respectfully deny that in such a case, memory
> buffering "isn't a big issue”.

well, 10s is a pretty long random access interval.

> 
> Now that I think about it, I guess there are more complicated strategies
> available that would reduce memory usage at the expense of repeated
> decoding.

which indeed QuickTime implemented around 10 years ago.

> E.g. in a first pass, decode forward and store every Nth frame.
> Then as you play backwards you need only redecode N-1 intermediate frames
> at time. I don't know whether HW decoder interfaces would actually let you
> implement that though...
> 
> What QuickTime got right was having a ToC approach to video so being able
>> to seek rapidly was possible without thrashing , whereas the stream
>> oriented approaches we are stuck with no wean knowing which bit of the file
>> to read to get the previous GOP is the hard part.
>> 
> 
> I don't understand. Can you explain this in more detail?

The movie file structure (and hence MP4) has a table-of-contents approach to 
file structure; each frame has its timestamps, file location, size, and 
keyframe-nature stored in compact tables in the head of the file.  This makes 
trick modes and so on easier; you’re not reading the actual video to seek for a 
keyframe, and so on.

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

On Tue, Sep 1, 2015 at 10:55 AM, David Singer  wrote:

>
> > On Sep 1, 2015, at 10:47 , Yay295  wrote:
> >
> > On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:
> > > On Sep 1, 2015, at 4:03 , Robert O'Callahan 
> wrote:
> > >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks 
> wrote:
> > >> QuickTime supports full variable speed playback and has done for well
> over
> > >> a decade. With bidirectionally predicted frames you need a fair few
> buffers
> > >> anyway, so generalising to full variable wait is easier than posters
> above
> > >> claim - you need to work a GOP at a time, but memory buffering isn't
> the
> > >> big issue these days.
> > >
> > > "GOP”?
> >
> > Group of Pictures.  Video-speak for the run between random access points.
> >
> > > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x
> 2
> > > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > > all has to stay in VRAM. I respectfully deny that in such a case,
> memory
> > > buffering "isn't a big issue”.
> >
> > well, 10s is a pretty long random access interval.
> >
> > There's no way to know the distance between keyframes though. The video
> could technically have only one keyframe and still work as a video.
>
> yes, but that is rare. There are indeed videos that don’t play well
> backward, or consume lots of memory and/or CPU, but most are fine.
>
> >
> > >> What QuickTime got right was having a ToC approach to video so being
> able
> > >> to seek rapidly was possible without thrashing , whereas the stream
> > >> oriented approaches we are stuck with no wean knowing which bit of
> the file
> > >> to read to get the previous GOP is the hard part.
> > >
> > > I don't understand. Can you explain this in more detail?
>

I explained the essential difference a while ago here:
http://lists.xiph.org/pipermail/vorbis-dev/2001-October/004846.html

The QuickTime file format defines movies that have tracks made of media;
the tracks are en edit list on the media; the media have the frame layout
information encoded.


> >
> > The movie file structure (and hence MP4) has a table-of-contents
> approach to file structure; each frame has its timestamps, file location,
> size, and keyframe-nature stored in compact tables in the head of the file.
> This makes trick modes and so on easier; you’re not reading the actual
> video to seek for a keyframe, and so on.
> >
> > I suppose the browser could generate this data the first time it reads
> through the video. It would use a lot less memory. Though that sounds like
> a problem for the browsers to solve, not the standard.
>
> There is no *generation* on the browser side; these tables are part of the
> file format.


Well, when it imports stream-oriented media it has to construct these in
memory, but they can be saved out again. I know that in theory this made
its way into the mp4 format, but I'm not sure how much of it is real.

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

On Tue, Sep 1, 2015 at 11:57 AM, David Singer  wrote:

>
> > On Sep 1, 2015, at 11:36 , Kevin Marks  wrote:
> > > I suppose the browser could generate this data the first time it reads
> through the video. It would use a lot less memory. Though that sounds like
> a problem for the browsers to solve, not the standard.
> >
> > There is no *generation* on the browser side; these tables are part of
> the file format.
> >
> > Well, when it imports stream-oriented media it has to construct these in
> memory, but they can be saved out again. I know that in theory this made
> its way into the mp4 format, but I'm not sure how much of it is real.
>
> Two different questions:
> a) do the QuickTime movie file format and the MP4 format contain these
> tables?  Yes.
> b) if I open another format, what happens?
>
> For case (a), the situation may be more nuanced if Movie Fragments are in
> use (you then get the tables for each fragment of the movie, though they
> are easily coalesced as they arrive).
>
> For case (b), classic QuickTime used to ‘convert to movie’ in memory,
> building the tables.  The situation is more nuanced on more recent engines.
>
> I think the point of the discussion is that one cannot dismiss trick modes
> such as reverse play as being unimplementable.


The other point for me is that given http://aomedia.org/ announcing plans
to create a new video file format to fix everything, that this time we
actually learn from this history and make one that is editable and seekable
again.

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 11:36 , Kevin Marks  wrote:
> > I suppose the browser could generate this data the first time it reads 
> > through the video. It would use a lot less memory. Though that sounds like 
> > a problem for the browsers to solve, not the standard.
> 
> There is no *generation* on the browser side; these tables are part of the 
> file format.
> 
> Well, when it imports stream-oriented media it has to construct these in 
> memory, but they can be saved out again. I know that in theory this made its 
> way into the mp4 format, but I'm not sure how much of it is real.

Two different questions:
a) do the QuickTime movie file format and the MP4 format contain these tables?  
Yes.
b) if I open another format, what happens?

For case (a), the situation may be more nuanced if Movie Fragments are in use 
(you then get the tables for each fragment of the movie, though they are easily 
coalesced as they arrive).

For case (b), classic QuickTime used to ‘convert to movie’ in memory, building 
the tables.  The situation is more nuanced on more recent engines.

I think the point of the discussion is that one cannot dismiss trick modes such 
as reverse play as being unimplementable. 

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:

> > On Sep 1, 2015, at 4:03 , Robert O'Callahan 
> wrote:
> >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks 
> wrote:
> >> QuickTime supports full variable speed playback and has done for well
> over
> >> a decade. With bidirectionally predicted frames you need a fair few
> buffers
> >> anyway, so generalising to full variable wait is easier than posters
> above
> >> claim - you need to work a GOP at a time, but memory buffering isn't the
> >> big issue these days.
> >
> > "GOP”?
>
> Group of Pictures.  Video-speak for the run between random access points.
>
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
>
> well, 10s is a pretty long random access interval.
>

There's no way to know the distance between keyframes though. The video
could technically have only one keyframe and still work as a video.


> >> What QuickTime got right was having a ToC approach to video so being
> able
> >> to seek rapidly was possible without thrashing , whereas the stream
> >> oriented approaches we are stuck with no wean knowing which bit of the
> file
> >> to read to get the previous GOP is the hard part.
> >
> > I don't understand. Can you explain this in more detail?
>
> The movie file structure (and hence MP4) has a table-of-contents approach
> to file structure; each frame has its timestamps, file location, size, and
> keyframe-nature stored in compact tables in the head of the file. This
> makes trick modes and so on easier; you’re not reading the actual video to
> seek for a keyframe, and so on.
>

I suppose the browser could generate this data the first time it reads
through the video. It would use a lot less memory. Though that sounds like
a problem for the browsers to solve, not the standard.

Re: [whatwg] VIDEO and pitchAdjustment

[David Singer]

> On Sep 1, 2015, at 10:47 , Yay295  wrote:
> 
> On Tue, Sep 1, 2015 at 11:30 AM, David Singer  wrote:
> > On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> >> On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:
> >> QuickTime supports full variable speed playback and has done for well over
> >> a decade. With bidirectionally predicted frames you need a fair few buffers
> >> anyway, so generalising to full variable wait is easier than posters above
> >> claim - you need to work a GOP at a time, but memory buffering isn't the
> >> big issue these days.
> >
> > "GOP”?
> 
> Group of Pictures.  Video-speak for the run between random access points.
> 
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
> 
> well, 10s is a pretty long random access interval.
> 
> There's no way to know the distance between keyframes though. The video could 
> technically have only one keyframe and still work as a video.

yes, but that is rare. There are indeed videos that don’t play well backward, 
or consume lots of memory and/or CPU, but most are fine.

>  
> >> What QuickTime got right was having a ToC approach to video so being able
> >> to seek rapidly was possible without thrashing , whereas the stream
> >> oriented approaches we are stuck with no wean knowing which bit of the file
> >> to read to get the previous GOP is the hard part.
> >
> > I don't understand. Can you explain this in more detail?
> 
> The movie file structure (and hence MP4) has a table-of-contents approach to 
> file structure; each frame has its timestamps, file location, size, and 
> keyframe-nature stored in compact tables in the head of the file. This makes 
> trick modes and so on easier; you’re not reading the actual video to seek for 
> a keyframe, and so on.
> 
> I suppose the browser could generate this data the first time it reads 
> through the video. It would use a lot less memory. Though that sounds like a 
> problem for the browsers to solve, not the standard.

There is no *generation* on the browser side; these tables are part of the file 
format.

David Singer
Manager, Software Standards, Apple Inc.

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Wed, Sep 2, 2015 at 5:30 AM, David Singer  wrote:

> On Sep 1, 2015, at 4:03 , Robert O'Callahan  wrote:
> > How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25
> fps,
> > keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
> > bytes = 4.32 GiB. Reading back those frames would kill performance so
> that
> > all has to stay in VRAM. I respectfully deny that in such a case, memory
> > buffering "isn't a big issue”.
>
> well, 10s is a pretty long random access interval.
>

It's easy to find sources on the Internet advising people to use 10s
keyframe intervals.

> Now that I think about it, I guess there are more complicated strategies
> > available that would reduce memory usage at the expense of repeated
> > decoding.
>
> which indeed QuickTime implemented around 10 years ago.
>

It appears that most platform and HW decoder interfaces are incompatible
with this strategy, so in practice implementing this across platforms is
still a big problem.

Nevertheless we can hope for that situation to improve, and negative
playback rates are implementable for some videos, so it makes sense to me
to leave negative playback rates in the spec.

The movie file structure (and hence MP4) has a table-of-contents approach
> to file structure; each frame has its timestamps, file location, size, and
> keyframe-nature stored in compact tables in the head of the file.  This
> makes trick modes and so on easier; you’re not reading the actual video to
> seek for a keyframe, and so on.
>

I think every important video container format has some kind of keyframe
directory.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Kevin Marks]

QuickTime supports full variable speed playback and has done for well over
a decade. With bidirectionally predicted frames you need a fair few buffers
anyway, so generalising to full variable wait is easier than posters above
claim - you need to work a GOP at a time, but memory buffering isn't the
big issue these days.
What QuickTime got right was having a ToC approach to video so being able
to seek rapidly was possible without thrashing , whereas the stream
oriented approaches we are stuck with no wean knowing which bit of the file
to read to get the previous GOP is the hard part.

On Fri, Aug 28, 2015 at 6:02 PM, Xidorn Quan  wrote:

> On Sat, Aug 29, 2015 at 8:27 AM, Robert O'Callahan 
> wrote:
> > On Sat, Aug 29, 2015 at 8:18 AM, James Ross 
> wrote:
> >
> >> Support is certainly poor; Internet Explorer/Trident and Edge both
> support
> >> negative playback rates on desktop (I haven’t tested mobile) but do so
> by
> >> simply showing the key frames as they are reached in reverse, in my
> testing.
> >
> > That's not so hard to implement, but it's also mostly useless since
> > keyframes are often several seconds apart or more.
>
> It could be useful for a few usecases like fast-backward. Windows
> Media Player does it this way.
>
> FWIW, QuickTime supports per-frame backward playback if you press and
> hold the left arrow. I guess they cannot guarantee the rate, which
> makes them require holding the key instead of providing a playback
> rate setting.
>
> - Xidorn
>

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
>
>>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>>
>> ...
>>
>>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
>> content.
>
> That's really helpful, thanks. Combined with Edge's keyframes-only support, 
> it sounds like we should probably leave the spec as it is.
>
> Do you have thoughts on a mozPreservesPitch equivalent?

Should we just standardize HTMLMediaElement.preservesPitch, perhaps?
Note that WebKit also has webkitPreservesPitch, but I removed it from
Blink because it didn't actually do anything in Chromium.

In both Gecko and WebKit it defaults to true. Is there anything else
worth knowing before writing the spec for this?

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Tue, Sep 1, 2015 at 8:02 PM, Kevin Marks  wrote:

> QuickTime supports full variable speed playback and has done for well over
> a decade. With bidirectionally predicted frames you need a fair few buffers
> anyway, so generalising to full variable wait is easier than posters above
> claim - you need to work a GOP at a time, but memory buffering isn't the
> big issue these days.
>

"GOP"?

How about a hard but realistic (IMHO) case: 4K video (4096 x 2160), 25 fps,
keyframe every 10s. Storing all those frames takes 250 x 4096 x 2160 x 2
bytes = 4.32 GiB. Reading back those frames would kill performance so that
all has to stay in VRAM. I respectfully deny that in such a case, memory
buffering "isn't a big issue".

Now that I think about it, I guess there are more complicated strategies
available that would reduce memory usage at the expense of repeated
decoding. E.g. in a first pass, decode forward and store every Nth frame.
Then as you play backwards you need only redecode N-1 intermediate frames
at time. I don't know whether HW decoder interfaces would actually let you
implement that though...

What QuickTime got right was having a ToC approach to video so being able
> to seek rapidly was possible without thrashing , whereas the stream
> oriented approaches we are stuck with no wean knowing which bit of the file
> to read to get the previous GOP is the hard part.
>

I don't understand. Can you explain this in more detail?

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Michael Enright]

On Fri, Aug 28, 2015 at 11:17 AM, Domenic Denicola  wrote:
> From:  Robert O'Callahan
>
>> According to the spec it should work, but it's very low priority for us and
>> implementing it would be very inefficient as Yay295 describes. So I don't
>> think it's going to happen in Firefox in the forseeable future.
>
> I was looking in to this yesterday and it seems like the spec places absolute 
> no limits on playbackRate. Am I right? This seems a bit bad.
>
> If nobody is supporting negative nor has any plans to, we should at least 
> consider throwing for negative. I guess we can leave the upper limit 
> unspecified, unless implementations all happen to agree on one.
>

I have implemented video element implementations which allowed the
video to play in reverse, and scripts that ran on such implementations
that allowed the user to ask for playback in reverse. If it is thought
that browser scripts need guidance on what speeds are possible for a
given server or video, then by all means go that route.

USE CASE: Implementing things like VidiPath or in some other fashion
implementing satellte STB or cable STB UI in HTML 5.

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/31/15, Michael Enright  wrote:
> On Fri, Aug 28, 2015 at 11:17 AM, Domenic Denicola  wrote:
>> From:  Robert O'Callahan
>>
>>> According to the spec it should work, but it's very low priority for us
>>> and
>>> implementing it would be very inefficient as Yay295 describes. So I
>>> don't
>>> think it's going to happen in Firefox in the forseeable future.
>>
>> I was looking in to this yesterday and it seems like the spec places
>> absolute no limits on playbackRate. Am I right? This seems a bit bad.
>>
>> If nobody is supporting negative nor has any plans to, we should at least
>> consider throwing for negative. I guess we can leave the upper limit
>> unspecified, unless implementations all happen to agree on one.
>>
>
> I have implemented video element implementations which allowed the
> video to play in reverse, and scripts that ran on such implementations
> that allowed the user to ask for playback in reverse. If it is thought
> that browser scripts need guidance on what speeds are possible for a
> given server or video, then by all means go that route.
>
> USE CASE: Implementing things like VidiPath or in some other fashion
> implementing satellte STB or cable STB UI in HTML 5.
>

USE CASE: https://www.youtube.com/watch?v=BtSdt1G6mcE
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

To be clear:

Everyone can imagine use cases for playing videos backward. However, so far the 
only statements we have about implementations are negative. My subthread was 
more concerned with making the spec reflect current reality. If you can 
convince implementers to support backward videos, then that's separate, and we 
can change the spec again.

Re: [whatwg] VIDEO and pitchAdjustment

[Eric Carlson]

> On Aug 31, 2015, at 12:04 PM, Domenic Denicola  wrote:
> 
> My subthread was more concerned with making the spec reflect current reality. 
> If you can convince implementers to support backward videos, then that's 
> separate, and we can change the spec again.
> 


  FWIW, Safari supports negative playback rates on the desktop and on iOS. 


> On Aug 27, 2015, at 11:02 AM, Garrett Smith  wrote:
> 
> Negative playbackRate, to watch videos backwards, currently crashes
> Safari 8


  The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE 
content.

eric

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: Eric Carlson [mailto:eric.carl...@apple.com]

>   FWIW, Safari supports negative playback rates on the desktop and on iOS.
>
> ...
>
>   The crash Garrett noted in Safari 8 is a bug that “only" happens with MSE
> content.

That's really helpful, thanks. Combined with Edge's keyframes-only support, it 
sounds like we should probably leave the spec as it is.

Do you have thoughts on a mozPreservesPitch equivalent?

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Sat, Aug 29, 2015 at 8:18 AM, James Ross ja...@james-ross.co.uk wrote:

 Support is certainly poor; Internet Explorer/Trident and Edge both support
 negative playback rates on desktop (I haven’t tested mobile) but do so by
 simply showing the key frames as they are reached in reverse, in my testing.


That's not so hard to implement, but it's also mostly useless since
keyframes are often several seconds apart or more.


 Firefox, Chrome and Safari on desktop and mobile don’t support negative
 values at all AFAICT. I have notes here suggesting that mobile platforms
 don’t even support positive rates other that 1.


I'm pretty sure mobile Firefox supports non-1 playback rates.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Domenic Denicola]

From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Robert 
O'Callahan

 According to the spec it should work, but it's very low priority for us and
 implementing it would be very inefficient as Yay295 describes. So I don't
 think it's going to happen in Firefox in the forseeable future.

I was looking in to this yesterday and it seems like the spec places absolute 
no limits on playbackRate. Am I right? This seems a bit bad.

If nobody is supporting negative nor has any plans to, we should at least 
consider throwing for negative. I guess we can leave the upper limit 
unspecified, unless implementations all happen to agree on one.

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Fri, Aug 28, 2015 at 12:17 PM, Domenic Denicola d...@domenic.me wrote:

 From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Robert
 O'Callahan
  According to the spec it should work, but it's very low priority for us
 and
  implementing it would be very inefficient as Yay295 describes. So I don't
  think it's going to happen in Firefox in the forseeable future.

 I was looking in to this yesterday and it seems like the spec places
 absolute no limits on playbackRate. Am I right? This seems a bit bad.

 If nobody is supporting negative nor has any plans to, we should at least
 consider throwing for negative. I guess we can leave the upper limit
 unspecified, unless implementations all happen to agree on one.


playbackRate should also probably be set to 0 when the video is paused,
seeing as they accomplish the same task. Or better yet redefine pause() to
be setting playbackRate to 0. You would of course then have to keep a note
of what the playbackRate was before pause() was called, but it would fix
the issue of playbackRate being set to zero, yet video.paused returning
false. So video.pause() would save the current playbackRate and then set
playbackRate to 0, and video.play() would restore the playbackRate saved by
pause(). video.paused could then just return (playbackRate ? false : true).

Re: [whatwg] VIDEO and pitchAdjustment

[Xidorn Quan]

On Sat, Aug 29, 2015 at 8:27 AM, Robert O'Callahan rob...@ocallahan.org wrote:
 On Sat, Aug 29, 2015 at 8:18 AM, James Ross ja...@james-ross.co.uk wrote:

 Support is certainly poor; Internet Explorer/Trident and Edge both support
 negative playback rates on desktop (I haven’t tested mobile) but do so by
 simply showing the key frames as they are reached in reverse, in my testing.

 That's not so hard to implement, but it's also mostly useless since
 keyframes are often several seconds apart or more.

It could be useful for a few usecases like fast-backward. Windows
Media Player does it this way.

FWIW, QuickTime supports per-frame backward playback if you press and
hold the left arrow. I guess they cannot guarantee the rate, which
makes them require holding the key instead of providing a playback
rate setting.

- Xidorn

Re: [whatwg] VIDEO and pitchAdjustment

[Yay295]

On Thu, Aug 27, 2015 at 12:02 PM, Garrett Smith dhtmlkitc...@gmail.com
wrote:

 Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


It would be much easier to re-encode the videos in reverse and provide two
versions of the video. Videos are made to be played in one direction, and
they use this assumption to improve their compression efficiency.
Basically, videos have two different types of frames; let's call them Full
frames and Partial frames. A Full frame contains all of the information
that a stand-alone image would have. If a video had only a single Full
frame, it would be able to display that image and show it for the length of
the video (incidentally, this is what a WebP image is). Partial frames, on
the other hand, do NOT have enough information to produce a full image.
They merely contain the information representing the difference between the
previous frame and the current frame. This obviously saves a lot of space
if the frames are similar. A video will contain a mixture of Full frames
and Partial frames, with many more Partial frames than Full frames (ex. 1
Full frame followed by 10 seconds of Partial frames (~240 partial frames)).
What this means is that when you play a video in the proper direction, it
must first decode a Full frame, but then it only has to decode Partial
frames (until the next Full frame). However, if you try to play a video in
reverse, it must decode a Full frame and ALL of the Partial frames up to
the current frame before it can display the current frame. To display the
frame before that, it must decode all of those frames again, and so on for
every frame of the video. You could of course store the decoded frames in
memory, but that will take a lot of space. 240 uncompressed frames at 720p
is over half a gigabyte of information. It's just not practical.

Re: [whatwg] VIDEO and pitchAdjustment

[Robert O'Callahan]

On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith dhtmlkitc...@gmail.com
wrote:

 It would be useful to have pitch adjustment for VIDEO element. There
 is playbackRate, to control playback speed — useful!* And there is
 vv.mozPreservesPitch, in Firefox, which can be set to false, so that
 pitch will adjust to the speed of the video, sort of like old analog
 gear (tapes and records).


We should standardize mozPreservesPitch. I'm embarrassed that we haven't
done so already.

Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


According to the spec it should work, but it's very low priority for us and
implementing it would be very inefficient as Yay295 describes. So I don't
think it's going to happen in Firefox in the forseeable future.

But variable pitch control it would be useful for music adjustments
 like over the mountain,  Black Star, Take your Whiskey Home, all
 originally recorded at half-step detuning (A=415), none of which match
 standard tuning commonly used in schools, industry, etc. Recordings
 of Baroque era music often use instruments tuned lower (and also
 different scale temperment, but that is a different issue). So it
 would be nice to adjust the pitch with a `pitchAdjustment` property,
 as a double, to adjust pitch in cents.


I'd prefer to focus on making sure that Web Audio's Audio Workers are
powerful enough for Web developers to implement this themselves.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 8/27/15, Robert O'Callahan rob...@ocallahan.org wrote:
 On Fri, Aug 28, 2015 at 6:02 AM, Garrett Smith dhtmlkitc...@gmail.com
 wrote:

 It would be useful to have pitch adjustment for VIDEO element. There
 is playbackRate, to control playback speed — useful!* And there is
 vv.mozPreservesPitch, in Firefox, which can be set to false, so that
 pitch will adjust to the speed of the video, sort of like old analog
 gear (tapes and records).


 We should standardize mozPreservesPitch. I'm embarrassed that we haven't
 done so already.

 Negative playbackRate, to watch videos backwards, currently crashes
 Safari 8; Firefox 40 says not implemented. I think it would be
 entertaining for example, to watch things like cars uncrashing. Should
 this work? And if so, shouldn't audio match the video speed for
 backwards playing? I think yes, and yes.


 According to the spec it should work, but it's very low priority for us and
 implementing it would be very inefficient as Yay295 describes. So I don't
 think it's going to happen in Firefox in the forseeable future.

 But variable pitch control it would be useful for music adjustments
 like over the mountain,  Black Star, Take your Whiskey Home, all
 originally recorded at half-step detuning (A=415), none of which match
 standard tuning commonly used in schools, industry, etc. Recordings
 of Baroque era music often use instruments tuned lower (and also
 different scale temperment, but that is a different issue). So it
 would be nice to adjust the pitch with a `pitchAdjustment` property,
 as a double, to adjust pitch in cents.


 I'd prefer to focus on making sure that Web Audio's Audio Workers are
 powerful enough for Web developers to implement this themselves.


I don't know enough about Web Audio to comment on that. Most of what
I'm using is on YouTube.  And there are some video issues that I'd
like to comment on because it's such a common situation to have audio
as video MP4's. So here's two video user scenarios:

Want hear the audio at the desired pitch, while simultaneously
watching the video at the desired speed, to learn it.

For fine-grained adjustment for music performances that are in between
keys (off by a few cents) or need to be transcribed.

This guy's cover of Over the Mountain is pretty good.
https://www.youtube.com/watch?v=Ze6h1t7Z734 Right now, I can get that
up a half step to standard tuning by increasing the speed and setting
mozPreservesPitch:—

var vv = document.querySelector(video);
vv.mozPreservesPitch = false;
vv.playbackRate = 1.05;

But it's a little sped up. Not ideal for learning! I'd rather slow
down it a bit. Enter the Amazing Slow Downer … does what I want.

OK, there's my two user scenarios with an example. Thanks for reading!
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com