*Summary*: MediaStreamTrackProcessor is the consuming end of a (video) MediaStreamTrack whereas VideoTrackGenerator is the producing end. Together they let sites process video in a worker by for instance drawing frames to an OffscreenCanvas, applying a transform, and reading them back out again.
Similar functionality is already available through the HTMLCanvasElement.captureStream main thread API but this approach has drawbacks. One is that requestAnimationFrame is throttled in background tabs, forcing sites to use various hacks to keep processing running when in the background. MediaStreamTrackProcessor and VideoTrackGenerator solves this problem by natively providing an API for direct VideoFrame access, with timestamp information. The new interfaces will be available only on DedicatedWorker and only for video MediaStreamTracks. MediaStreamTrackHandle is a smaller unit than a MediaStreamTrack that one can transfer to a worker for use with MediaStreamTrackProcessor while control of the track remains on the main thread. *Bug*: Transferable tracks: bug 1991619 <https://bugzilla.mozilla.org/show_bug.cgi?id=1991619>MediaStreamTrackHandle: bug 2074270 <https://bugzilla.mozilla.org/show_bug.cgi?id=2074270> MediaStreamTrackProcessor: bug 1991617 <https://bugzilla.mozilla.org/show_bug.cgi?id=1991617> VideoTrackGenerator: bug 1991618 <https://bugzilla.mozilla.org/show_bug.cgi?id=1991618> *Specification*: Transferable tracks: w3c/mediacapture-extensions <https://w3c.github.io/mediacapture-extensions/#transferable-mediastreamtrack> MediaStreamTrackProcessor/VideoTrackGenerator/MediaStreamTrackHandle: w3c/mediacapture-transform <https://w3c.github.io/mediacapture-transform/> *Platform Coverage*: All platforms. *Preference*: Nothing has landed yet, but let's use: media.mediastreamtrack.transferable.enabled media.mediastreamtrackprocessor.enabled media.videotrackgenerator.enabled *DevTools Bug*: MediaStreamTracks have no special handling in devtools. *Extensions Bug*: MediaStreamTracks have no special handling in extensions. *Use Counter*: TBD *Standards-Positions Discussion*: Transferable tracks: 1044 <https://github.com/mozilla/standards-positions/issues/1044> MediaStreamTrackProcessor/VideoTrackGenerator: 1078 <https://github.com/mozilla/standards-positions/issues/1078> *Other Browsers*: - Blink: Shipped an older form of MediaStreamTrackProcessor <https://chromestatus.com/feature/5499415634640896> (exposed on Window instead of DedicatedWorker) in Chromium 94. Per-spec VideoTrackGenerator is tracked in chromium/40058895 <https://issues.chromium.org/issues/40058895> and the support on DedicatedWorker in chromium/509846181 <https://issues.chromium.org/issues/509846181>. - WebKit: Shipped in Safari 18. *web-platform-tests*: Transferable tracks: in part on medicapture-stream <https://wpt.fyi/results/mediacapture-streams?label=master&label=experimental&aligned&q=transfer>, in part on mediacapture-extensions <https://wpt.fyi/results/mediacapture-extensions?label=master&label=experimental&aligned&q=worker> . MediaStreamTrackProcessor/VideoTrackGenerator: mediacapture-insertable-streams <https://wpt.fyi/results/mediacapture-insertable-streams?label=master&label=experimental&aligned>, and we'll add more as necessary. Not filed yet. Andreas Pehrson WebRTC Media Engineer Slack: @pehrsons Mozilla -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAOX9k0%3DZwsGixVkj2aHCpMTVEUtwie4y1OMD_iJE_djVf4tjjw%40mail.gmail.com.
