Hi Dan,
Thanks for reaching out. Here is some feedback based on the web codecs
explainer (let me know if there is a more detailed proposal).
- I am generally in favour of trying to unify the various media
pipelines. It is always sad when one feature ships in the video
streaming pipeline, and not automatically to the RTC one (or the
reverse).
- Efficient processing of raw video frames (whatever its source:
camera, RTC, video streaming) seems like a really useful area to work
on.
- Existing APIs (video/image tags, MSE, WebRTC, recording API...)
already provide features that make use of codecs. It should be clear
what the goals/key benefits of directly exposing codecs are.
- Any media pipeline should be off the JS main thread, by default.
This does not seem guaranteed by the proposal.
- Providing deep access to codecs (in terms of capabilities,
observability of timing of operations...) requires careful thinking of
how much fingerprinting this ends up creating and how the processing
model will ensure to keep the whole API fingerprinting neutral.
- A codec implementation used for RTC may differ significantly from a
codec implementation used for recording/MSE. With this proposal, a web
page could try to use for RTC purposes a hardware codec dedicated to
recording/MSE. The results would be disastrous. That will probably
require extensive testing by web developers to ensure their scripts
are working in a wide variety of devices. At some point, that might
require supporting APIs to properly discover and setup
encoders/decoders for the various uses. This might further add to both
complexity and fingerprinting.
- The complexity behind WebRTC, MSE or the media recorder API is not
to be neglected. There might be drawbacks in solving these issues at
the JS level instead of the browser level. I am for instance uncertain
that the level of complexity of a WebRTC pipeline can best be solved
by JS.
- Some code might best stay in control of the browser. Related to
WebCodecs is the insertable stream proposal which could allow
deploying end-to-end encryption quickly to RTC with a JS-only
solution. A JS solution leaves full control to web pages and limits
the ability for user agents to upgrade such security mechanisms like
they can do for other security mechanisms like TLS/DTLS.
As a background information, I would also note the effort done lately
in WebKit to move parts of media processing out of processes running
JavaScript.
Hope this helps,
Y
Le jeu. 30 avr. 2020 à 00:53, Dan Sanders <sande...@chromium.org> a écrit :
>
> Hello,
>
> I'm reaching out to see if WebKit would like to weigh in on the
> WebCodecs WICG proposal:
> https://discourse.wicg.io/t/webcodecs-proposal/3662.
>
> The WebCodecs API enables web developers to instantiate codecs
> (audio/video encoders/decoders) and use them to process individual
> frames.
>
> There is a related proposal for image decoders; it enables access to
> individual animation frames:
> https://discourse.wicg.io/t/proposal-imagedecoder-api-extension-for-webcodecs/4418.
>
> An implementation of these APIs is being developed in Chromium.
>
>
> Thank you,
>
> Dan Sanders
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
