May I please ask two questions? 1) Has any thought been given to exposing the segmentation result - a grayscale mask defining the foreground (person) vs. the background?
Building this functionality on top of technologies like MediaPipe, Tensorflow.js, Wasm, etc. uses the web platform's existing low-level, composable primitives to provide a range of capabiilties. On the web today, in addition to background blur, MediaPipe and Meet offer many background replacement options, which are arguably more privacy-preserving than just blurring. If the segmentation result were provided, the web application could implement these effects on top of it. It's understood that this API enables the use of custom IP cores which can deliver a result with lower power. At the same time, it's less flexible than what can be done on the web today. 2) Have the MediaPipe or Meet teams been asked for input? MediaPipe and Meet pioneered background effects in videoconferencing on the web. They're conspicuously absent from the "Stakeholder Feedback/Opposition" section of https://github.com/riju/backgroundBlur/blob/main/explainer.md . Thanks, -Ken On Thu, Jun 23, 2022 at 1:34 AM Bhaumik, Rijubrata < [email protected]> wrote: > We are going to first implement this on Windows. “Real” support is > dependent on driver support. If on your client machine, the camera app can > do it, then this API can also do it. > I don’t have a Mac, but there’s platform support > <https://bugs.chromium.org/p/chromium/issues/detail?id=1281408>, so we > can collaborate once the initial Blink CLs lands. > > *From:* 'François Beaufort 🇫🇷' via blink-dev <[email protected]> > *Sent:* Thursday, June 23, 2022 10:51 AM > *To:* Yoav Weiss <[email protected]> > *Cc:* Bhaumik, Rijubrata <[email protected]>; > [email protected]; Guido Urdaneta <[email protected]>; Harald > Alvestrand <[email protected]>; Youenn Fablet <[email protected]>; Bernard > Aboba <[email protected]>; Jan-Ivar Bruaroey @Mozilla < > [email protected]> > *Subject:* Re: [blink-dev] Intent to Prototype : Background Blur API. > > > > > > Thank you for starting this effort! > > Quick question: Will this feature be supported on all six Blink platforms > (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Not just > exposing backgroundBlur properties but also it "real" support. > > > > On Thu, Jun 23, 2022 at 9:26 AM Yoav Weiss <[email protected]> wrote: > > Exciting work!! > > > > On Wed, Jun 22, 2022 at 11:46 PM Bhaumik, Rijubrata < > [email protected]> wrote: > > Contact emails > > [email protected], [email protected] > Explainer > > https://github.com/riju/backgroundBlur/blob/main/explainer.md > Specification > > > https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support > Summary > > Background Blur API gives web developers a choice to use the native > platform's API for background segmentation. Background Blur has become one > of the most used features on Video conferencing apps like Teams, Meet, > Zoom, Webex, etc. We want give Web apps similar capabilities as their > native counterparts, leveraging the same platform APIs and to delight users > without completely relying on ML frameworks like TensorFlow.js, Mediapipe, > WASM libraries or cloud based solutions. > > > Blink component > > Blink>ImageCapture > <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EImageCapture> > Motivation A vast majority of communication these days happens on our > client devices. During video meetings, participants are usually aware of > how they look and what their environment (usually their home) is revealing > to the audience. Most folks, especially ones without a dedicated office > space would be inclined to hide messy rooms with pets and kids. Video > meetings like face to face meetings are important for non-verbal > communication but participants would rather focus on the important subject > by removing the distractions in the background and prevent any accidental > snafus. Microsoft says > <https://www.microsoft.com/en-ww/microsoft-365/business-insights-ideas/resources/how-custom-backgrounds-keep-the-focus-on-you> > in > a 38 minute conference call, 13 minutes are wasted dealing with > distractions and interruptions. Background Blur goes a long way to cutting > down those disruptions. Zoom says > <https://support.zoom.us/hc/en-us/articles/360061468611-Using-blurred-background>- > "*When a custom virtual background is unavailable or not suiting your > needs, but you still want to maintain some privacy with regards to your > surroundings, the blur background option can be a great alternative. This > option simply blurs the background of your video, obscuring exactly who or > what is behind you. It's great for hiding a cluttered dorm room, taking a > meeting in a coffee shop, or just keeping things professional.*" . In > fact, NCSC (National Cyber Security Centre UK) suggests using background > Blur or a background image > <https://www.ncsc.gov.uk/guidance/video-conferencing-services-security-guidance-organisations> > for > staff meetings to add a degree of personal privacy. On the Web, due to a > lack of a standardized API for Background Blur and widespread demand, > developers have no options but to use ML frameworks like Tensorflow.js and > other WASM libraries to satisfy their customers. This Background Blur API > gives developers a choice to use the native platform's API. Initial > public proposal > > https://github.com/w3c/mediacapture-extensions/pull/49 > https://github.com/w3c/mediacapture-extensions/pull/61 > Search tags > > background blur <https://chromestatus.com/features#tags:background%20blur> > , webrtc <https://chromestatus.com/features#tags:webrtc> > TAG review > > > > May be a good idea to file for a TAG review early. > > > > TAG review status > > Pending > Risks > > > Interoperability and Compatibility > > > > *Gecko*: No signal, but spec LGTM ed by Jan-Ivar (Mozilla). > https://github.com/mozilla/standards-positions/issues/658 > > *WebKit*: No signal but Apple co-author of explainer / PR. > https://lists.webkit.org/pipermail/webkit-dev/2022-June/032311.html > > *Web developers*: No signals > > *Other signals*: Almost every video-conference app has background blur > feature. > > WebView application risks > > *Does this intent deprecate or change behavior of existing APIs, such that > it has potentially high risk for Android WebView-based applications?* > > > Debuggability > > > Is this feature fully tested by web-platform-tests > <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> > ? > > Yes ( as part of CL ) > Flag name > > MediaCaptureBackgroundBlur > > Requires code in //chrome? > > False > Tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=1338665 > Estimated milestones > > No milestones specified > > > Link to entry on the Chrome Platform Status > > https://chromestatus.com/feature/5077577782263808 > > This intent message was generated by Chrome Platform Status > <https://chromestatus.com/>. > > > > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191E86DD684B0D86A12B0EEF0B29%40PH0PR11MB5191.namprd11.prod.outlook.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191E86DD684B0D86A12B0EEF0B29%40PH0PR11MB5191.namprd11.prod.outlook.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWu4kMKsspZYAchmqjLSvXKb5HT1wen0E5JqjBSyG8onw%40mail.gmail.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWu4kMKsspZYAchmqjLSvXKb5HT1wen0E5JqjBSyG8onw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPpwU5KKz%2BPuBdyTBP9w_W%3D5ny9jTErBVHpJbpYCVZmDmgLGbQ%40mail.gmail.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPpwU5KKz%2BPuBdyTBP9w_W%3D5ny9jTErBVHpJbpYCVZmDmgLGbQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F9937507ED7BDAC101FDF0B59%40PH0PR11MB5191.namprd11.prod.outlook.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F9937507ED7BDAC101FDF0B59%40PH0PR11MB5191.namprd11.prod.outlook.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMYvS2fLZbd827sE9jtkcjtHTxTiP1ymAN9j6s9sgPXP2WMj0w%40mail.gmail.com.
