On 3/24/2014 3:11 AM, Gene Lian wrote:
Hi Media folks,
I already read through Media Capture and Streams API [1], but I'd like to
clarify myself about how to leverage the current getUserMedia() to retrieve
the video/audio tracks from the specific input like HDMI or AV for example,
which is needed in our coming TV product.
Could anyone please give us some directions about the following questions
when you're available? Thank you!
1. getUserMedia() will return a MediaStream based on the user's selection
on the pop-up prompt, but why the MediaStream can contain multiple
MediaStreamTracks where each of them could represent different media
sources [2]?
For example, when the user selects the "Webcam", it makes sense that
the returned MediaStream will contain one MediaStreamTrack for the
video stream and one MediaStreamTrack for the audio stream in terms of
a webcam, but how can it happen to contain either multiple videos or
multiple audios for one webcam? I mean why can getVideoTracks() or
getAudioTracks() returns an array?
Check the Media Capture W3 spec and mailing list. There's no reason why
getUserMedia() couldn't return multiple cameras as one stream if the
user so chooses (though no current implementation allows this).
getUserMedia() could return a media stream sourced from a file or even
remote streaming media, and some of those will have multiple audio or
video tracks.
Also note that getUserMedia is not the only source of a MediaStream.
See Web Audio, and also mozCaptureStreamUntilEnded() (which we're going
to push to be adopted or some equivalent, along with sourcing from a
Canvas).
And, a MediaStream can be constructed from multiple tracks from other
MediaStreams. For example, two MediaStreams from getUserMedia() - one
front camera, one back camera - can be combined into a single
MediaStream and sent over a PeerConnection. You could send them
separately, but then the two streams wouldn't be synchronized (putting
things in one MediaStream implies synchronization).
That said - we don't currently really support multiple tracks of a given
type in a MediaStream, but we do plan to.
2. It seems the pop-up prompt and the user's selection are always needed
to specify the media source. Can we directly specify it when calling
getUserMedia()? Say, the user only wants to retrieve the MediaStream
through the specific HDMI device.
See the spec - we plan to allow specifying specific devices via opaque
identifiers. Chrome has this now IIRC. This allows an application to
default to "same camera as last time".
3. Follow #2, the Constraints may help us do so but it is aimed to apply
the constraints (source settings) on the media source, instead of
selecting the media source. Does setting sourceId in the constrain
exactly work for this purpose [3][5]?
yes.
4. Follow #3, does the deviceId in MediaDeviceInfo [4] match the sourceId
at [5]? That would be helpful if we can set the deviceId in the
constraint to getUserMedia() as mentioned above.
Without re-checking the spec, I believe the answer is yes.
5. It looks a bit weird to me that we can set constraints for calling
getUserMedia() and select the media source through the prompt at the
same time. What if the setting doesn't work for the media source the
user selects? Say, set the video width/height in the constraints but
the user chooses microphone by the prompt.
The prompt is supposed to provide you with options that match any
mandatory constraints set by the app. Note that considerable debate
around constraints (and optional vs mandatory) is still on-going - W3
teleconference on Thursday about this. If the app sets an unusable
constraint, bad app. Don't do that. The user can't choose a microphone
if the request was for video, note.
6. The getMediaDevices() can help us retrieve all the devices which are
currently available on the device [6]. One question here: does the menu
of the prompt rely on this function in practice?
They both rely on the same code.
7. Follow #6, how can the getMediaDevices() reflect some dynamic changes
whenever a device is hooked up or off at run-time? GamepadEvent [7]
might be a similar way to handle this, which is fired on the window,
but it means we have to add one event for each media source under the
window. Can we unify this kind of event under NavigatorUserMedia [6],
which can then be differentiated by the SourceTypeEnum [8]? IMO, I'd
suggest adding the following event handlers:
interface NavigatorUserMedia {
void getMediaDevices (MediaDeviceInfoCallback resultCallback);
attribute EventHandler onsourceconnected; // SourceEvent
attribute EventHandler onsourcedisconnected; // SourceEvent
};
interface SourceEvent : Event {
readonly attribute SourceTypeEnum type;
...
};
We need to support "hot-plug" of devices while the request is up; we
have a design for doing so but haven't detailed it yet. That's focused
on internal use; no W3 discussion of change notification has occurred
for public APIs IIRC (and certainly no API for it yet).
8. Can we extend the SourceTypeEnum [8] by adding "hdmi", "av" and so on?
Or the enum list has to be some formal spec defined at somewhere?
Well, it is in the formal spec you reference. See 14.1 for the IANA
registration info. hdmi might be possible to add. av seems very
unlikely and too vague as to the intent - and likely would just cause
failures if things used it. Best advice for apps is "avoid using any
constraints. If you do, make them optional. Really. Only in odd cases
do you want mandatory constraints"
In fact, hdmi is one of the rare cases where a source type might
actually be useful - generally it's not.
[1] http://dev.w3.org/2011/webrtc/editor/getusermedia.html
[2] http://dev.w3.org/2011/webrtc/editor/getusermedia.html#idl-def-MediaStream
[3]
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#idl-def-MediaStreamConstraints
[4]
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#idl-def-MediaDeviceInfo
[5]
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#track-property-registrations
[6] http://dev.w3.org/2011/webrtc/editor/getusermedia.html#navigatorusermedia
[7]
https://dvcs.w3.org/hg/gamepad/raw-file/default/gamepad.html#gamepadevent-interface
[8]
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#idl-def-SourceTypeEnum
Sorry for asking so many questions at one shot. I believe these are all
we have before moving forward. Looking forward to you experts' feedback. :)
--
Randell Jesup, Mozilla
_______________________________________________
dev-media mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-media
