Re: Intent to ship: media-capabilities

[Jean-Yves Avenard]

It appears that I hadn’t provided all the information earlier…

So here it is again:

Summary:
Media Capabilities allow for web sites to better determine what content to 
serve to the end user.
Currently a media element offers the canPlayType method 
(https://html.spec.whatwg.org/multipage/media.html#dom-navigator-canplaytype-dev)
 to determine if a container/codec can be used. But the answer is limited as a 
maybe/probably type answer.

It gives no ability to determine if a particular resolution can be played 
well/smoothly enough or be done in a power efficient manner (e.g. will it be 
hardware accelerated).

This has been a particular problem with sites such as YouTube that serves VP9 
under all circumstances even if the user agent won't play it well (VP9 is 
mostly done via software decoding and is CPU itensive). This has forced us to 
indiscriminately disable VP9 altogether).
For YouTube to know that VP9 could be used for low resolution but not high-def 
ones would allow them to select the right codec from the start.

Chrome has shipped it a while ago now and talking to several partners 
(including YouTube, Netflix, Facebook etc) , Media Capabilities support has 
been the number one request.

Bug: This issue is tracked in bugzilla 1409664  
(https://bugzilla.mozilla.org/show_bug.cgi?id=1409664)

Link to standard: The proposed spec is available at 
https://wicg.github.io/media-capabilities/
Platform coverage: It will be available for all platform, and exposed to all 
sites including insecure (http)
Estimated or target release: 63
Preference behind which this will be implemented: the feature is controllable 
via media.media-capabilities.enabled
Is this feature enabled by default in sandboxed iframes? If not, is there a 
proposed sandbox flag to enable it? If allowed, does it preserve the current 
invariants in terms of what sandboxed iframes can do?
DevTools bug: No particular requirements for additional devtools
Do other browser engines implement this? Chrome has shipped this since late 2017
web-platform-tests: Phttp://w3c-test.org/media-capabilities/

We do not enable the Screen Media-Capabilities extension due spec issues (in 
particular https://github.com/WICG/media-capabilities/issues/89), additionally, 
we have no way at present to implement those.



smime.p7s
Description: S/MIME cryptographic signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Jean-Yves Avenard]

Hi

There has been some concerns about some parts of the spec, in particular the 
one extending the Screen interface.

The plan now is to keep the Screen extensions disabled by default and to enable 
the remaining parts, related purely to the playing and encoding capabilities on.

This is tracked in bug 1480190

JY

> On 4 Jul 2018, at 2:16 am, Jean-Yves Avenard  wrote:
> 
> Hi
> 
> The code is now in central and in the last nightly.
> 
> It's currently disabled by default behind the pref 
> media.media-capabilities.enabled
> 
> The bug tracking fingerprinting concerns is done in 
> https://bugzilla.mozilla.org/show_bug.cgi?id=1461454 
> 
> 
> Feel free to enable it and watch videos in YouTube. On mac in particular it 
> would allow to re-enable the free vp9 codec (which has been disabled due to 
> performance reason)
> 
> Kind regards
> Jean-Yves



smime.p7s
Description: S/MIME cryptographic signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Jean-Yves Avenard]

Hi

The code is now in central and in the last nightly.

It's currently disabled by default behind the pref
media.media-capabilities.enabled

The bug tracking fingerprinting concerns is done in
https://bugzilla.mozilla.org/show_bug.cgi?id=1461454

Feel free to enable it and watch videos in YouTube. On mac in particular it
would allow to re-enable the free vp9 codec (which has been disabled due to
performance reason)

Kind regards
Jean-Yves

On Mon, May 14, 2018 at 5:19 PM, Jean-Yves Avenard 
wrote:

> Media Capabilities allow for web sites to better determine what content to
> serve to the end user.
> Currently a media element offers the canPlayType method (
> https://html.spec.whatwg.org/multipage/media.html#dom-
> navigator-canplaytype-dev) to determine if a container/codec can be used.
> But the answer is limited as a maybe/probably type answer.
>
> It gives no ability to determine if a particular resolution can be played
> well/smoothly enough or be done in a power efficient manner (e.g. will it
> be hardware accelerated).
>
> This has been a particular problem with sites such as YouTube that serves
> VP9 under all circumstances even if the user agent won't play it well (VP9
> is mostly done via software decoding and is CPU itensive). This has forced
> us to indiscriminately disable VP9 altogether).
> For YouTube to know that VP9 could be used for low resolution but not
> high-def ones would allow them to select the right codec from the start.
>
> This issue is tracked in bugzilla 1409664  (https://bugzilla.mozilla.org/
> show_bug.cgi?id=1409664)
>
> The proposed spec is available at https://wicg.github.io/
> media-capabilities/
>
> Chrome has shipped it a while ago now and talking to several partners
> (including YouTube, Netflix, Facebook etc) , Media Capabilities support has
> been the number one request.
>
> We intend to implement and ship this API very soon.
>
> Early comment and feedback will be welcome.
>
> Kinds regards
> Jean-Yves
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Karl Tomlinson]

Steven Englehardt writes:

> While it may have been
> theoretically possible for all trackers to gather statistics on video
> playback for each configuration, the only scripts that could practically
> carry out those attacks without degrading user experience would have been
> video providers. This will be especially true if browsers start blocking
> autoplay by default (https://bugzilla.mozilla.org/show_bug.cgi?id=1376321),
> since users will never interact with media elements from fingerprinting
> scripts. [...]

> If autoplay is eventually blocked by default could we gate the response of
> this API on user interaction with the media element?

Note that current playback blocking work is focused on media
elements that produce audio.  Video-only elements would not be
affected.

The blocking is also gated by interaction with the document,
rather than any particular element.

(There may be other non-default behaviors available.)
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Randell Jesup]

>If my understanding is correct, Media Capabilities does expose quite a
>larger fingerprinting surface in practice. While it may have been
>theoretically possible for all trackers to gather statistics on video
>playback for each configuration, the only scripts that could practically
>carry out those attacks without degrading user experience would have been
>video providers. This will be especially true if browsers start blocking
>autoplay by default (https://bugzilla.mozilla.org/show_bug.cgi?id=1376321),
>since users will never interact with media elements from fingerprinting
>scripts. With the Media Capabilities API, it seems that a script like
>fingerprintjs2 (https://github.com/Valve/fingerprintjs2) could run through
>a big list of types/codecs and retrieve device information regarding
>smoothness and energy efficiency with relatively little overhead?

Probably so, yes.  We could reduce but not eliminate the exposure by
rate-limiting requests (perhaps even on a sliding scale, allowing a
small number before delays are introduced).  This is likely insufficient
as a mitigation, however.

>If autoplay is eventually blocked by default could we gate the response of
>this API on user interaction with the media element?

That might be possible, but if so it should be discussed in the spec and
how to get "real" data after user interaction.  (Perhaps giving fake
data until user interaction, but then one needs to warn developers about
this, and how to get real data when interaction occurs reliably.)

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Boris Zbarsky]

On 5/14/18 3:18 PM, Jean-Yves Avenard wrote:

The most obvious choice considered was to provide identical information to what 
the existing canPlayType information provide: that is not providing extra 
details.


OK.  All I'm saying is that this needs to be sorted out before we ship.


I would invite you to submit such bug and concern you have on the wicg site:
https://github.com/wicg/media-capabilities/issues


I can do that, sure.  Figured I'd check whether these issues had been 
considered yet first.  Filed 
https://github.com/WICG/media-capabilities/issues/82



Having said that, with hardware decoders, typically whatever you may be doing 
has no impact on performance: it’s a dedicated circuit (even if for some 
there’s a limit on how many decoders can be used at the same time).


Sure; the question is what happens when the decoders are not hardware.


4) For the "change" event on Screen, does that apply to any property defined in 
any specification, not just the properties defined in this specification?  That would be 
a pretty significant monkeypatch in its own right.  It would be better if whatever 
specifications define properties also define what events fire if those properties change.



I’m not sure I understand your question. onChange and the change event is only 
defined for the Screen interface 
(https://drafts.csswg.org/cssom-view/#the-screen-interface).


Yes...  For which properties of that interface?  For example, should it 
fire for changes to availWidth?  Changes to pixelDepth?



Or you’re suggesting that as the MediaCapabilities Screen extension is only 
about gamut and luminance, each should get its own event so that future 
extension to the Screen interface do no conflict?


I don't have a strong opinion on that, though having a single "change" 
event mean "yeah, one of these N things changed" is not that great; then 
you have to re-poll all those N things to figure out which one changed. 
But my real objection is that the MediaCapabilities spec is currently 
saying that changes to _any_ property on Screen, defined by _any_ 
specification, necessitate a change event to be fired.  That is a pretty 
severe constraint on what other specifications can expose on Screen, no? 
 In particular exposing anything that the OS doesn't notify on changes 
for and is somewhat expensive (whatever that means) to query the OS for 
would suddenly become a non-starter.


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Steven Englehardt]

> > On 14 May 2018, at 6:47 pm, Tom Ritter  wrote:
> >
> > It seems like this will reveal a lot of information about the user's
> > hardware. Does the Resist Fingerprinting preference disable the API or
> > report standardized results? If not, can we get that bug on file (and
> > if it's easy, point out exactly where we would want to add the 'if()
> > return false'?)
> >
> > -tom
>
> This is a concern that has been raised previously, and one that you can
> ultimately get with existing APIs, but those are typically after the fact,
> and by then it’s already too late to allow the user to have a decent media
> playback experience
>
> Existing canPlayType can tell you if we support a particular codec or not.
> During playback, we already expose various metrics (starting from bug
> https://bugzilla.mozilla.org/show_bug.cgi?id=580531) this became an
> official spec, to determine if the content plays well : number of frames
> dropped, number of frames decoded, how many were painted etc...
>
> As such MediaCapabilities doesn’t expose much more than what someone can
> already gather over time with what’s already existing.
>

Not a domain expert, so I'd like to make sure I understand the difference
between what was possible with `canPlayType` + Media Statistics and what is
possible with Media Capabilities. Please correct me if I have it wrong!
Previously, a script wanting to fingerprint the user would have had to (1)
probe content type/codec support with `canPlayType`, (2) play videos using
each of these configurations, (3) measure and compute metrics for each
configuration using the media statistics API. With Media Capabilities, a
script still needs to probe each content type/codec pair (i.e., they can't
retrieve a list of all supported codecs?), and will receive a
classification of whether the device supports it, is expected to play the
video smoothly, and is expected to play the video in a power efficient way.

If my understanding is correct, Media Capabilities does expose quite a
larger fingerprinting surface in practice. While it may have been
theoretically possible for all trackers to gather statistics on video
playback for each configuration, the only scripts that could practically
carry out those attacks without degrading user experience would have been
video providers. This will be especially true if browsers start blocking
autoplay by default (https://bugzilla.mozilla.org/show_bug.cgi?id=1376321),
since users will never interact with media elements from fingerprinting
scripts. With the Media Capabilities API, it seems that a script like
fingerprintjs2 (https://github.com/Valve/fingerprintjs2) could run through
a big list of types/codecs and retrieve device information regarding
smoothness and energy efficiency with relatively little overhead?

If autoplay is eventually blocked by default could we gate the response of
this API on user interaction with the media element?


> To get around fingerprinting, at the user’s choice, the obvious work
> around would be to report that everything is always supported, will always
> do so smoothly with great battery savings. This is something we already do
> for the existing apps. The user will end up with a poor video experience
> however. As it will typically be served content not always adapted to his
> machine capabilities.
>

It would be great to have the "fingerprint-proof" mode described in more
detail in the spec. That will go a long way to making sure the protection
implemented in RFP / Tor Browser doesn't break sites and provides the best
possible performance given the restrictions.


> Providing a way to ensure the user will get a good video experience is
> paramount IMHO. Watching video on their web browser is what people do the
> most…
>
> JY
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Jean-Yves Avenard]

Hi

> On 14 May 2018, at 6:53 pm, Boris Zbarsky  wrote:
> 
> On 5/14/18 11:19 AM, Jean-Yves Avenard wrote:
>> The proposed spec is available at https://wicg.github.io/media-capabilities/ 
>> 
> 
> I have some questions about this spec and our implementation:
> 
We’re at an early stage in the implementation, and assuming the concerns of 
some, I wanted to present it early on.

> 1)  What are the fingerprinting implications?  What effect, if any, do our 
> "resist fingerprinting" preferences have on our API implementation here?  The 
> spec tries to address this but as usualy mostly handwaves around it.

The most obvious choice considered was to provide identical information to what 
the existing canPlayType information provide: that is not providing extra 
details.
so if canPlayType reports that "video/webm; codecs=vp9” is supported, then so 
will MediaCapabilities, but providing no difference then according to the 
resolution or the bitrate specified.
It is currently possible with canPlayType to query much deeper level of 
information, in particular bitrate, colorspace, HDR support, codec level etc… 
We haven’t fully implemented those because as canPlayType is a synchronous API, 
doing so properly with our asynchronous backend is hard.

> 
> 2) It looks to me that given a MediaCapabilitiesInfo there is no way to 
> figure out what configuration it represents.  Should there be such a way?  It 
> seems like it would make it simpler to deal with asking for the capabilities 
> for several configurations at once and then examining the results if you 
> don't have to keep track of which returned promise corresponds to which 
> passed-in configuration.  Doubly so if you Promise.race things (though why 
> one would do that in this case is not so clear to me).
> 
> Note that even the example in section 5.1 of the spec gets this wrong: it 
> uses result.contentType, but "result" is a MediaCapabilitiesInfo and doesn't 
> have a .contentType property.

I would invite you to submit such bug and concern you have on the wicg site: 
https://github.com/wicg/media-capabilities/issues

Or I can do so if you prefer.


> 3) The booleans in MediaCapabilitiesInfo (apart from "supported") seem rather 
> vaguely defined.  As a concrete example, if I am on 4-core (+ hyperthreading) 
> "desktop"-level system with nothing running except the video, "smooth" should 
> clearly be set to true.  Should it still be set to true on the same hardware 
> but in a situation where I am heavily swapping and my load average is 150?  
> This is a bit of a caricature, but it seems to me that if people are going to 
> treat this as a _reliable_ signal then it needs to be more clearly spelled 
> out what things it does or does not take into account.

this is an issue I’ve been raising frequently, that there’s no way to determine 
if the capabilities change over time: receiving a notification when such 
temporary workload occurs would be of benefit.
The spec isn’t set in stone, and I’m hoping that a new event could be 
dispatched on the media element to indicate that the capabilities have changed.

Having said that, with hardware decoders, typically whatever you may be doing 
has no impact on performance: it’s a dedicated circuit (even if for some 
there’s a limit on how many decoders can be used at the same time).

> 
> 4) For the "change" event on Screen, does that apply to any property defined 
> in any specification, not just the properties defined in this specification?  
> That would be a pretty significant monkeypatch in its own right.  It would be 
> better if whatever specifications define properties also define what events 
> fire if those properties change.
> 

I’m not sure I understand your question. onChange and the change event is only 
defined for the Screen interface 
(https://drafts.csswg.org/cssom-view/#the-screen-interface).
Or you’re suggesting that as the MediaCapabilities Screen extension is only 
about gamut and luminance, each should get its own event so that future 
extension to the Screen interface do no conflict?

JY

smime.p7s
Description: S/MIME cryptographic signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Jean-Yves Avenard]

Hi

> On 14 May 2018, at 6:47 pm, Tom Ritter  wrote:
> 
> It seems like this will reveal a lot of information about the user's
> hardware. Does the Resist Fingerprinting preference disable the API or
> report standardized results? If not, can we get that bug on file (and
> if it's easy, point out exactly where we would want to add the 'if()
> return false'?)
> 
> -tom

This is a concern that has been raised previously, and one that you can 
ultimately get with existing APIs, but those are typically after the fact, and 
by then it’s already too late to allow the user to have a decent media playback 
experience

Existing canPlayType can tell you if we support a particular codec or not.
During playback, we already expose various metrics (starting from bug 
https://bugzilla.mozilla.org/show_bug.cgi?id=580531) this became an official 
spec, to determine if the content plays well : number of frames dropped, number 
of frames decoded, how many were painted etc...

As such MediaCapabilities doesn’t expose much more than what someone can 
already gather over time with what’s already existing.

There are various ways we can build the Media Capabilities answer: collecting 
past metrics and build up a dictionary, or make assumptions based on the 
decoders (e.g. we know a hardware h264 decoder will always be smooth and power 
efficient).

To get around fingerprinting, at the user’s choice, the obvious work around 
would be to report that everything is always supported, will always do so 
smoothly with great battery savings. This is something we already do for the 
existing apps. The user will end up with a poor video experience however. As it 
will typically be served content not always adapted to his machine capabilities.

Providing a way to ensure the user will get a good video experience is 
paramount IMHO. Watching video on their web browser is what people do the most…

JY

smime.p7s
Description: S/MIME cryptographic signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Boris Zbarsky]

On 5/14/18 11:19 AM, Jean-Yves Avenard wrote:

The proposed spec is available at https://wicg.github.io/media-capabilities/ 



I have some questions about this spec and our implementation:

1)  What are the fingerprinting implications?  What effect, if any, do 
our "resist fingerprinting" preferences have on our API implementation 
here?  The spec tries to address this but as usualy mostly handwaves 
around it.


2) It looks to me that given a MediaCapabilitiesInfo there is no way to 
figure out what configuration it represents.  Should there be such a 
way?  It seems like it would make it simpler to deal with asking for the 
capabilities for several configurations at once and then examining the 
results if you don't have to keep track of which returned promise 
corresponds to which passed-in configuration.  Doubly so if you 
Promise.race things (though why one would do that in this case is not so 
clear to me).


Note that even the example in section 5.1 of the spec gets this wrong: 
it uses result.contentType, but "result" is a MediaCapabilitiesInfo and 
doesn't have a .contentType property.


3) The booleans in MediaCapabilitiesInfo (apart from "supported") seem 
rather vaguely defined.  As a concrete example, if I am on 4-core (+ 
hyperthreading) "desktop"-level system with nothing running except the 
video, "smooth" should clearly be set to true.  Should it still be set 
to true on the same hardware but in a situation where I am heavily 
swapping and my load average is 150?  This is a bit of a caricature, but 
it seems to me that if people are going to treat this as a _reliable_ 
signal then it needs to be more clearly spelled out what things it does 
or does not take into account.


4) For the "change" event on Screen, does that apply to any property 
defined in any specification, not just the properties defined in this 
specification?  That would be a pretty significant monkeypatch in its 
own right.  It would be better if whatever specifications define 
properties also define what events fire if those properties change.


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[Tom Ritter]

It seems like this will reveal a lot of information about the user's
hardware. Does the Resist Fingerprinting preference disable the API or
report standardized results? If not, can we get that bug on file (and
if it's easy, point out exactly where we would want to add the 'if()
return false'?)

-tom

On Mon, May 14, 2018 at 10:19 AM, Jean-Yves Avenard
 wrote:
> Media Capabilities allow for web sites to better determine what content to 
> serve to the end user.
> Currently a media element offers the canPlayType method 
> (https://html.spec.whatwg.org/multipage/media.html#dom-navigator-canplaytype-dev
>  
> )
>  to determine if a container/codec can be used. But the answer is limited as 
> a maybe/probably type answer.
>
> It gives no ability to determine if a particular resolution can be played 
> well/smoothly enough or be done in a power efficient manner (e.g. will it be 
> hardware accelerated).
>
> This has been a particular problem with sites such as YouTube that serves VP9 
> under all circumstances even if the user agent won't play it well (VP9 is 
> mostly done via software decoding and is CPU itensive). This has forced us to 
> indiscriminately disable VP9 altogether).
> For YouTube to know that VP9 could be used for low resolution but not 
> high-def ones would allow them to select the right codec from the start.
>
> This issue is tracked in bugzilla 1409664  
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1409664 
> )
>
> The proposed spec is available at https://wicg.github.io/media-capabilities/ 
> 
>
> Chrome has shipped it a while ago now and talking to several partners 
> (including YouTube, Netflix, Facebook etc) , Media Capabilities support has 
> been the number one request.
>
> We intend to implement and ship this API very soon.
>
> Early comment and feedback will be welcome.
>
> Kinds regards
> Jean-Yves
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to ship: media-capabilities

[James Graham]

On 14/05/2018 16:19, Jean-Yves Avenard wrote:

Media Capabilities allow for web sites to better determine what content to 
serve to the end user.
Currently a media element offers the canPlayType method 
(https://html.spec.whatwg.org/multipage/media.html#dom-navigator-canplaytype-dev 
)
 to determine if a container/codec can be used. But the answer is limited as a 
maybe/probably type answer.

It gives no ability to determine if a particular resolution can be played 
well/smoothly enough or be done in a power efficient manner (e.g. will it be 
hardware accelerated).

This has been a particular problem with sites such as YouTube that serves VP9 
under all circumstances even if the user agent won't play it well (VP9 is 
mostly done via software decoding and is CPU itensive). This has forced us to 
indiscriminately disable VP9 altogether).
For YouTube to know that VP9 could be used for low resolution but not high-def 
ones would allow them to select the right codec from the start.

This issue is tracked in bugzilla 1409664  
(https://bugzilla.mozilla.org/show_bug.cgi?id=1409664 
)

The proposed spec is available at https://wicg.github.io/media-capabilities/ 


Chrome has shipped it a while ago now and talking to several partners 
(including YouTube, Netflix, Facebook etc) , Media Capabilities support has 
been the number one request.



What is the testing situation for this feature? Do we have 
web-platform-tests?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform