Re: [whatwg] metadata attribute for media
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 17/02/13 05:48 AM, Nils Dagsson Moskopp wrote: If one cares to that extent, and is already handling format differences, dealing with vendor variation on top isn't that much more effort. I disagree, strongly. Ok, thanks for the feedback. Do you like my subsequent proposal better? http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-January/038705.html -r -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJRImnnAAoJEEcAD3uxRB3vHBwH/14a91L4oN6nktTttTRvQQFf hmtz52nAO7BdInpG68fRACgLVsz2eT+eWakAAMuzh7T57NH9ZNCSWp1Rk/AM3BXD akx6QXQuk9eOwuRlk+BIVIUaKVFyod9D133BdTjGqF6/cVmn7emmIm9BWB0sxXrE 3CPuALqs+07ExHbyB/UsE5BWqsJRccJHg9QnJwXd4aMtHbIg37KAcfgN4aYVTHQV B1I95Uh1Ron2LUCNI9P+Pm8WPQGDcj0cuwFcueRMYy2O26a1xHisOew/l3Cz46Ex Oq6gAX3Lz/y5LxFhnnlPIeBDhBE9bB0r4jqOtY3FNA+dcUruKI769nIw5WtTmd4= =zJ6M -END PGP SIGNATURE-
Re: [whatwg] metadata attribute for media
Ralph Giles gi...@mozilla.com schrieb am Tue, 11 Dec 2012 17:23:38 -0800: On 12-12-11 4:58 PM, Ian Hickson wrote: […] I don't think we should have an open-ended API without fixed names, because that is a recipe for an interoperability disaster. I agree it would have interoperability issues. My own implementation experience is that the easy thing to do is to mirror whatever representation your playback framework offers, which can result in per-platform differences as well as per-media-format (and per tagging application, etc.). Just doing “the easiest thing” when you are a content-emitter (e.g. a software developer just mirroring what the platform provides) does make it easier for the sender while making it harder for the receiver – it creates “computational dark energy” that has to be consumed somehow at the receiving side, as someone has to map all of these platform-specific conventions onto common semantics. I rather not have a world in which one needs even more JavaScript blobs just to provide basic functionality cross-browser. That said, I'm not convinced this is an issue given the primary use-case, which is pretty much that web content wants to do more sophisticated things with the metadata than the user-agent's standardized parsing allows. If one cares to that extent, and is already handling format differences, dealing with vendor variation on top isn't that much more effort. I disagree, strongly. -- Nils Dagsson Moskopp // erlehmann http://dieweltistgarnichtso.net
Re: [whatwg] metadata attribute for media
On 12-12-11 5:23 PM, Ralph Giles wrote:
That said, I'm not convinced this is an issue given the primary
use-case, which is pretty much that web content wants to do more
sophisticated things with the metadata than the user-agent's
standardized parsing allows. If one cares to that extent, and is already
handling format differences, dealing with vendor variation on top isn't
that much more effort.
Robert O'Callahan argued (off-list) that there is a significant
difference. Exposing what the media framework supplies for metadata,
multiplies the established format fragmentation by per-platform
differences. That's not something we want to do if we can avoid it, and
we can avoid it by doing our own tag normalization as part of the
exposed API.
We feel that's more important than the extended and custom tag use case,
which is still addressible by direct parsing in javascript, etc.
I don't intend to continue with implementation of the raw metadata api,
and instead will focus on mapping everthing to a standard set of
attributes. To that proposal I've added a 'tracknumber' attribute since
this is imporant for sorting resources in media player applications.
interface Metadata {
readonly attribute DOMString title;
readonly attribute DOMString artist;
readonly attribute DOMString album;
readonly attribute long tracknumber;
readonly attribute Blob? artwork;
};
partial interface MediaElement {
Metadata metadata;
};
-r
Re: [whatwg] metadata attribute for media
On Wed, Dec 12, 2012 at 2:23 PM, Ralph Giles gi...@mozilla.com wrote: That said, I'm not convinced this is an issue given the primary use-case, which is pretty much that web content wants to do more sophisticated things with the metadata than the user-agent's standardized parsing allows. If one cares to that extent, and is already handling format differences, dealing with vendor variation on top isn't that much more effort. I disagree. Vendors proliferate more than formats. A Web application may well have some knowledge about the formats of its resources, but we don't want to bind it to particular platforms or UAs. Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] metadata attribute for media
On Thu, 20 Sep 2012, Ralph Giles wrote: Back in June, I proposed[1] a new attribute to get metadata tag data out of media resources. I've done an experimental implementation of this which is now in the Firefox Aurora (alpha) channel[2] and Nightly development builds. The method is media.mozGetMetadata() and it returns a new object each time, whose properties are key:value pairs representing the metadata tags from the file. Aurora supports this for Ogg Vorbis (.ogg) streams. Nightly supports Opus (.opus) files as well. Right now the method returns whatever set of tags happen to be in the media stream, at least if they can be interpreted as unicode text. There are several things I'd like to fix about this. Media formats generally have rules about how to map various tags to a standard vocabulary. Right now, web content has to implement those rules, but the user-agent is in a better position to do so. Also, a number of Mozilla (C++) developers I've talked to prefer a fixed schema for metadata, rather than the more dynamic approach. So, I'd like to do a new method which returns an object with a fixed schema we can describe in idl, representing standard fields derived from the dublic core tag set. The raw metadata would be available under the old method for advanced applications. Some metadata is also per-track, rather than per-stream, so it makes sense to have these calls available on track objects as well as media objects. The most essential tags are dc:creator and dc:title. That covers almost all player use cases. For per-track metadata, the language is perhaps also useful for selection purposes. Are there any other tags folks would like to see in the initial proposal? [1] http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-June/036352.html [2] available from aurora.mozilla.org This seems reasonable. If there's implementor interest from other vendors, and if there's a spec that I can point to for how to get the metadata out of the various formats, then I'd be happy to add it to the spec. I don't want to be the one to maintain the mapping from media formats to metadata schema, because this isn't my area of expertise, and it isn't trivial work. I don't think we should have an open-ended API without fixed names, because that is a recipe for an interoperability disaster. We'd just end up having to define the mapping later when major yet poorly-tested sites started depending on particular mappings of popular UAs. On Mon, 26 Nov 2012, Ralph Giles wrote: On 12-09-27 1:44 AM, Philip Jägenstedt wrote: I'm skeptical that all that we want from ID3v2 or common VorbisComment tags can be mapped to Dublin Core, it seems better to define mappings directly from the underlying format to the WebIDL interface. You're right. Indeed. Given the open-endedness of metadata contained in actual media resources, I'm personally a bit skeptical that there's something we could add to the Web platform that would be better than just letting authors pass that metadata out-of-band using any representation they like, but what use cases are you trying to cover here? Two use cases I'm trying to address: - A web application presents some view of a media library. If the libray resides on a server, then yes, the server-side component of the app can parse, cache, and deliver the metadata out-of-band. But the library could also be local, in which case the webapp must do its own parsing, e.g. from a list of blob urls returned by the file api. - An author wants to display the embedded track title and artist name with simple scripting on a webpage. For music in particular this seems reasonable. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] metadata attribute for media
On 12-12-11 4:58 PM, Ian Hickson wrote: This seems reasonable. Thanks for the feedback. Anyone else? :-) I don't want to be the one to maintain the mapping from media formats to metadata schema, because this isn't my area of expertise, and it isn't trivial work. Good point. This would need to be standardized for the fixed-schema proposal, at least for the formats commonly supported by the HTML media element. The Web Ontology working group has done some work here, as Silvia mentioned. I don't think we should have an open-ended API without fixed names, because that is a recipe for an interoperability disaster. I agree it would have interoperability issues. My own implementation experience is that the easy thing to do is to mirror whatever representation your playback framework offers, which can result in per-platform differences as well as per-media-format (and per tagging application, etc.). That said, I'm not convinced this is an issue given the primary use-case, which is pretty much that web content wants to do more sophisticated things with the metadata than the user-agent's standardized parsing allows. If one cares to that extent, and is already handling format differences, dealing with vendor variation on top isn't that much more effort. We could say that user-agents should represent the metadata dictionaries as directly as possible, and to match the tag names from the schema spec when that's not possible. -r
Re: [whatwg] metadata attribute for media
On 12-11-26 4:18 PM, Ralph Giles wrote:
interface HTMLMediaElement {
...
object getMetadata();
};
After the metadataloaded event fires, this method would return a new
object containing a copy of the metadata read from the resource, in
whatever format the decoder implementation supplies. It would be up to
the caller to do any semantic interpretation. The same method should be
present on AudioTrack, VideoTrack, (TextTrack?) and Image elements.
The prefixed version of this in Firefox is documented in the 'Methods'
section of https://developer.mozilla.org/en-US/docs/DOM/HTMLMediaElement
-r
Re: [whatwg] metadata attribute for media
On 12-09-27 1:44 AM, Philip Jägenstedt wrote:
I'm skeptical that all that we want from ID3v2 or common VorbisComment
tags can be mapped to Dublin Core, it seems better to define mappings
directly from the underlying format to the WebIDL interface.
You're right.
Given the open-endedness of metadata contained in actual media
resources, I'm personally a bit skeptical that there's something we
could add to the Web platform that would be better than just letting
authors pass that metadata out-of-band using any representation they
like, but what use cases are you trying to cover here?
Two use cases I'm trying to address:
- A web application presents some view of a media library. If the libray
resides on a server, then yes, the server-side component of the app can
parse, cache, and deliver the metadata out-of-band. But the library
could also be local, in which case the webapp must do its own parsing,
e.g. from a list of blob urls returned by the file api.
- An author wants to display the embedded track title and artist name
with simple scripting on a webpage.
One of the goals of html media was to make video and audio as simple to
include as images. User agents are generally parsing this metadata
anyway; exposing it is straightforward, and greatly simplifies the two
tasks above.
In any case, the media.mozGetMetadata() method I described earlier is
available in Firefox 17, released last week, with support for Vorbis
tags in .ogg files. Firefox 18, now in beta, adds support for .opus
files as well. Here's an example:
https://people.xiph.org/~giles/2012/metadata.html
You should see 'Title', 'Album', etc. below the controls if your browser
supports mozGetMetadata().
We're continuing to implement this interface for other formats we support.
I still think it's useful to define both this 'raw' metadata interface,
returning just the data out of the file, and a more structured metadata
object with standardized tag names. I've left off implementing the
second one for lack of feedback on what the basic tags should be, and
how useful they are. There certainly wasn't consensus here.
So, what do you think of these two proposals:
1. Define the 'raw' getMetadata method in an unprefixed form:
interface HTMLMediaElement {
...
object getMetadata();
};
After the metadataloaded event fires, this method would return a new
object containing a copy of the metadata read from the resource, in
whatever format the decoder implementation supplies. It would be up to
the caller to do any semantic interpretation. The same method should be
present on AudioTrack, VideoTrack, (TextTrack?) and Image elements.
2. Define a parsed metadata attribute with some standard tags:
interface Metadata {
readonly attribute DOMString title;
readonly attribute DOMString artist;
readonly attribute DOMString album;
readonly attribute Blob? artwork;
};
interface MediaElement {
...
Metadata metadata;
};
So you could say something as simple as
img.src = audio.metadata.artwork;
to display the cover art embedded in a download single. DOMString
attributes would have the value of an empty string if the underlying
data isn't available. These four attributes are the absolute minimum, I
think, for the use cases above. More could be added later as usage of
the api evolves. For example: date, publisher, tracknumber, tracktotal,
description, genre, sort-artist, sort-title, copyright, license, url.
If having both media.getMetadata() (raw) and media.metadata is
confusing, the first proposal could be named media.getRawMetadata() as well.
Does it make sense to include more technical metadata here? For example:
samplerate, channels, duration, width, height, framerate, aspect-ratio.
Firefox currently has prefixed properties for the number of channels and
the audio sample rate, and including these in the metadata interface
would let us deprecate the prefixed versions. On the other hand,
properties like duration, width, and height are available directly on
media elements today, so maybe it makes more sense to do the same for
the others.
In that case, do we want the indirection of the Metadata inferface?
Saying 'video.title' or 'img.src = audio.artwork' instead is shorter.
Feedback welcome,
-r
Re: [whatwg] metadata attribute for media
On Fri, 21 Sep 2012 01:32:19 +0200, Ralph Giles gi...@mozilla.com wrote: Back in June, I proposed[1] a new attribute to get metadata tag data out of media resources. I've done an experimental implementation of this which is now in the Firefox Aurora (alpha) channel[2] and Nightly development builds. The method is media.mozGetMetadata() and it returns a new object each time, whose properties are key:value pairs representing the metadata tags from the file. Aurora supports this for Ogg Vorbis (.ogg) streams. Nightly supports Opus (.opus) files as well. Right now the method returns whatever set of tags happen to be in the media stream, at least if they can be interpreted as unicode text. There are several things I'd like to fix about this. Media formats generally have rules about how to map various tags to a standard vocabulary. Right now, web content has to implement those rules, but the user-agent is in a better position to do so. Also, a number of Mozilla (C++) developers I've talked to prefer a fixed schema for metadata, rather than the more dynamic approach. So, I'd like to do a new method which returns an object with a fixed schema we can describe in idl, representing standard fields derived from the dublic core tag set. The raw metadata would be available under the old method for advanced applications. Some metadata is also per-track, rather than per-stream, so it makes sense to have these calls available on track objects as well as media objects. The most essential tags are dc:creator and dc:title. That covers almost all player use cases. For per-track metadata, the language is perhaps also useful for selection purposes. Are there any other tags folks would like to see in the initial proposal? I'm skeptical that all that we want from ID3v2 or common VorbisComment tags can be mapped to Dublin Core, it seems better to define mappings directly from the underlying format to the WebIDL interface. As an example, the MusicBrainz schema has both a track artist (e.g. Queen feat. George Michael) and an album artist (e.g. Queen) and two kinds of titles (track title and recording title) which doesn't seem to benefit from being passed through dc:creator and dc:title. Given the open-endedness of metadata contained in actual media resources, I'm personally a bit skeptical that there's something we could add to the Web platform that would be better than just letting authors pass that metadata out-of-band using any representation they like, but what use cases are you trying to cover here? -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] metadata attribute for media
On Tue, Jun 12, 2012 at 7:53 AM, Ralph Giles gi...@mozilla.com wrote:
Recently, we've been considering adding a 'tags' or 'metadata' attribute
to HTML media elements in Firefox, to allow webcontent access to
metadata from the playing media resource. In particular we're interested
in tag data like creator, title, date, and so on.
My recollection is that this has been discussed a number of times in the
past, but there was never suffient motivation to support the interface.
Our particular motivation here is webapps that present a media file
library. While it's certainly possible to parse the tag data out
directly with javascript, it's more convenient if the HTML media element
does so, and the underlying platform decoder libraries usually provide
this data already.
As such I wanted to raise the issue here and get design feedback and
levels of interest for other user agents.
Here's a first idea:
partial interface HTMLMediaElement {
readonly attribute object tags;
};
Accessing media.tags provides an object with a key: value data, for example:
{
'title': 'My Movie',
'creator': 'This User',
'date': '2012-06-18',
'license': 'http://creativecommons.org/licenses/by-nc-sa/'
}
The keys may need to be filtered, since the files can contain things
like base64-encoded cover art, which makes the object prohibitively
large. The keys may need to be mapped to some standard scheme (i.e.
dublic core) since vocabularies vary from format to format.
This is nice because it's easy to access, can be simply enumerated,
and extensible. Which is helpful when if gets added the img for exif data.
Did you know that the W3C media annotations WG has specified such an
API in http://www.w3.org/TR/2011/WD-mediaont-api-1.0-2022/#api-description
. Essentially, their suggestion is to add the following IDL functions:
void getMediaProperty (DOMString[] propertyNames, PropertyCallback
successCallback, ErrorCallback errorCallback, optional DOMString
fragment, optional DOMString sourceFormat, optional DOMString
language);
void getOriginalMetadata (DOMString sourceFormat, MetadataCallback
successCallback, ErrorCallback errorCallback);
I actually think their API is too complicated and prefer your simple
approach. Returning a JSON object also allows hierarchical tags to be
returned in a structured way, which is nice. You lose the
normalisation that the W3C media ann WG has worked on across different
media resources, but that normalisation can always be done on top of
the JSON objects that your API returns.
Cheers,
Silvia.
