Effects that are not supported by the camera driver need to be done in software like the Meet effects pipeline or Breakout Box processing.
tor. 10. nov. 2022, 18:54 skrev Vivek Wamorkar < [email protected]>: > Greetings, > Thanks for starting this effort. > From the spec and some of the answers, I think the proposal indicates that > the actual blur is applied on the source of the frames i.e. Camera (or > capturer)... > > I am wondering how would we support the following use cases: > 1. Support for the image (or other media) as a background > 2. If I have started a blurred stream toward the remote participants > (in the call), and would like to go to settings to try different background > images before I finalize on one. In this case I may need 2 streams (one to > continue using Blur, and 2nd try image) > 3. If I need to provide custom (user-uploaded) images to the > background. > > Please let me know, how these scenarios will be addressed or if could > throw some light on how the chromium would implement these. > > Thank you for your help > Regards > Vivek > > > On Monday, July 4, 2022 at 3:18:37 PM UTC+5:30 [email protected] wrote: > >> Hi Bhaumik, >> >> Thanks for the precisions! Let me rephrase to verify my understanding: >> "Background blur is applied by request or constraint. In the >> first implementation only stream sources like cameras obtained by >> GetUserMedia support blurring. This is completely unidirectional, from the >> device to the platform. Getting a stream from another source such as >> CanvasCaptureMediaStreamTrack will simply not support the capability. >> Therefore there is no way to send custom crafted bits into the native APIs, >> and in general no content from the internet flows into the native APIs." >> >> Is that accurate? >> >> Cheers, >> Arthur >> >> >> On Thu, Jun 30, 2022 at 3:32 PM Bhaumik, Rijubrata <[email protected]> >> wrote: >> >>> Hi Arthur, >>> >>> Thanks for the questions, I will add more info in the Explainer >>> regarding Privacy in Security >>> <https://github.com/riju/backgroundBlur/blob/main/explainer.md#security-considerations> >>> . >>> >>> Replies inline - >>> >>> >>> >>> *From:* Arthur Hemery <[email protected]> >>> *Sent:* Wednesday, June 29, 2022 1:17 PM >>> *To:* blink-dev <[email protected]> >>> *Cc:* Bhaumik, Rijubrata <[email protected]>; François Beaufort < >>> [email protected]>; [email protected] <[email protected]>; >>> [email protected] <[email protected]>; Guido Urdaneta < >>> [email protected]>; Harald Alvestrand <[email protected]>; Youenn >>> Fablet <[email protected]>; Bernard Aboba <[email protected]>; >>> Jan-Ivar Bruaroey @Mozilla <[email protected]>; Kenneth Russell < >>> [email protected]> >>> *Subject:* Re: [blink-dev] Intent to Prototype : Background Blur API. >>> >>> >>> >>> Hi there! >>> >>> We discussed this yesterday in the OWP Security&Privacy review and had a >>> couple of questions/remarks: >>> - Using underlying platform APIs is fine as long as all providers are >>> aware that they are going to receive random content from the internet. It's >>> important because they are often more privileged than Chrome. We should >>> strongly fuzz these API to verify that they meet the bar. >>> >>> *[Bhaumik, Rijubrata**] The only source for Background Blur API is >>> client-side camera via getUserMedia() and the native API is going to do the >>> entire processing on the client. The native APIs does in-stream correction. >>> Only then is the "blurred" stream sent to the internet.* >>> >>> >>> - From reading the spec it seems possible to obtain a MediaStreamTrack >>> that contains arbitrary data and to apply background blur to it. Is that >>> correct? If that's the case, then it makes fuzzing even more important. >>> >>> *[Bhaumik, Rijubrata] **Actually, the implementation [**CL* >>> <https://chromium-review.googlesource.com/c/chromium/src/+/3714592>*] >>> does not itself apply background blur but it instead requests camera >>> (drivers) to apply background blur so that the browser gets video frames >>> whose background is blurred. I am not sure if it’s possible for the browser >>> to retrieve unblurred video frames and try to apply background blur >>> constraint to (frames originating from a canvas) those video frames, but >>> this implementation does not do that. * >>> >>> >>> - With this change we know via constraints when the system has already >>> blurred the image right? I imagine you get this information only after >>> you've queried a MediaStreamTrack? >>> >>> *[Bhaumik, Rijubrata] **Yes, it is possible to know via the >>> MediaStreamTrack’s settings the current background blurring is in effect >>> (whether it is due to a system wide default or due to a constraints applied >>> by a web application). The MediaStreamTrack’s settings are only available >>> after a web application has requested to get a MediaStream and the user has >>> granted that.* >>> >>> >>> >>> Thanks! >>> Arthur >>> >>> On Wednesday, June 29, 2022 at 10:16:55 AM UTC+2 [email protected] >>> wrote: >>> >>> Thanks Ken, >>> >>> >>> >>> 1. segmentation result - a grayscale mask : Yes, Harald (Google) did >>> comment on this, something that MediaPipe uses. We discussed that MASK >>> would be more important in Background Replacement scenarios. Since no >>> platform APIs support Background Replacement right now, we decided to >>> move >>> it to future work >>> <https://github.com/riju/backgroundBlur/blob/main/explainer.md#non-goals> >>> . >>> 2. Meet for input : Sorry, fixed that in the explainer >>> >>> <https://github.com/riju/backgroundBlur/blob/main/explainer.md#user-research>. >>> I am reaching out to Meet again. In general, most of the stakeholders >>> were >>> supportive and thought it’s definitely worth prototyping. I will try to >>> add >>> more quotes/ feedback from others. >>> >>> >>> >>> >>> >>> *From:* Ken Russell <[email protected]> >>> *Sent:* Saturday, June 25, 2022 1:46 AM >>> *To:* Bhaumik, Rijubrata <[email protected]> >>> *Cc:* François Beaufort 🇫🇷 <[email protected]>; Yoav Weiss < >>> [email protected]>; [email protected]; Guido Urdaneta < >>> [email protected]>; Harald Alvestrand <[email protected]>; Youenn >>> Fablet <[email protected]>; Bernard Aboba <[email protected]>; >>> Jan-Ivar Bruaroey @Mozilla <[email protected]> >>> *Subject:* Re: [blink-dev] Intent to Prototype : Background Blur API. >>> >>> >>> >>> May I please ask two questions? >>> >>> >>> >>> 1) Has any thought been given to exposing the segmentation result - a >>> grayscale mask defining the foreground (person) vs. the background? >>> >>> >>> >>> Building this functionality on top of technologies like >>> MediaPipe, Tensorflow.js, Wasm, etc. uses the web platform's existing >>> low-level, composable primitives to provide a range of capabiilties. On the >>> web today, in addition to background blur, MediaPipe and Meet offer many >>> background replacement options, which are arguably more privacy-preserving >>> than just blurring. If the segmentation result were provided, the web >>> application could implement these effects on top of it. >>> >>> >>> >>> It's understood that this API enables the use of custom IP cores which >>> can deliver a result with lower power. At the same time, it's less >>> flexible than what can be done on the web today. >>> >>> >>> >>> 2) Have the MediaPipe or Meet teams been asked for input? MediaPipe and >>> Meet pioneered background effects in videoconferencing on the web. They're >>> conspicuously absent from the "Stakeholder Feedback/Opposition" section of >>> https://github.com/riju/backgroundBlur/blob/main/explainer.md . >>> >>> >>> >>> Thanks, >>> >>> >>> >>> -Ken >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Jun 23, 2022 at 1:34 AM Bhaumik, Rijubrata < >>> [email protected]> wrote: >>> >>> We are going to first implement this on Windows. “Real” support is >>> dependent on driver support. If on your client machine, the camera app can >>> do it, then this API can also do it. >>> I don’t have a Mac, but there’s platform support >>> <https://bugs.chromium.org/p/chromium/issues/detail?id=1281408>, so we >>> can collaborate once the initial Blink CLs lands. >>> >>> *From:* 'François Beaufort 🇫🇷' via blink-dev <[email protected]> >>> *Sent:* Thursday, June 23, 2022 10:51 AM >>> *To:* Yoav Weiss <[email protected]> >>> *Cc:* Bhaumik, Rijubrata <[email protected]>; [email protected]; >>> Guido Urdaneta <[email protected]>; Harald Alvestrand <[email protected]>; >>> Youenn Fablet <[email protected]>; Bernard Aboba <[email protected]>; >>> Jan-Ivar Bruaroey @Mozilla <[email protected]> >>> *Subject:* Re: [blink-dev] Intent to Prototype : Background Blur API. >>> >>> >>> >>> >>> >>> Thank you for starting this effort! >>> >>> Quick question: Will this feature be supported on all six Blink >>> platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? >>> Not just exposing backgroundBlur properties but also it "real" support. >>> >>> >>> >>> On Thu, Jun 23, 2022 at 9:26 AM Yoav Weiss <[email protected]> wrote: >>> >>> Exciting work!! >>> >>> >>> >>> On Wed, Jun 22, 2022 at 11:46 PM Bhaumik, Rijubrata < >>> [email protected]> wrote: >>> >>> Contact emails >>> >>> [email protected], [email protected] >>> Explainer >>> >>> https://github.com/riju/backgroundBlur/blob/main/explainer.md >>> Specification >>> >>> >>> https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support >>> Summary >>> >>> Background Blur API gives web developers a choice to use the native >>> platform's API for background segmentation. Background Blur has become one >>> of the most used features on Video conferencing apps like Teams, Meet, >>> Zoom, Webex, etc. We want give Web apps similar capabilities as their >>> native counterparts, leveraging the same platform APIs and to delight users >>> without completely relying on ML frameworks like TensorFlow.js, Mediapipe, >>> WASM libraries or cloud based solutions. >>> >>> >>> Blink component >>> >>> Blink>ImageCapture >>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EImageCapture> >>> Motivation A vast majority of communication these days happens on our >>> client devices. During video meetings, participants are usually aware of >>> how they look and what their environment (usually their home) is revealing >>> to the audience. Most folks, especially ones without a dedicated office >>> space would be inclined to hide messy rooms with pets and kids. Video >>> meetings like face to face meetings are important for non-verbal >>> communication but participants would rather focus on the important subject >>> by removing the distractions in the background and prevent any accidental >>> snafus. Microsoft says >>> <https://www.microsoft.com/en-ww/microsoft-365/business-insights-ideas/resources/how-custom-backgrounds-keep-the-focus-on-you> >>> in >>> a 38 minute conference call, 13 minutes are wasted dealing with >>> distractions and interruptions. Background Blur goes a long way to cutting >>> down those disruptions. Zoom says >>> <https://support.zoom.us/hc/en-us/articles/360061468611-Using-blurred-background>- >>> "*When a custom virtual background is unavailable or not suiting your >>> needs, but you still want to maintain some privacy with regards to your >>> surroundings, the blur background option can be a great alternative. This >>> option simply blurs the background of your video, obscuring exactly who or >>> what is behind you. It's great for hiding a cluttered dorm room, taking a >>> meeting in a coffee shop, or just keeping things professional.*" . In >>> fact, NCSC (National Cyber Security Centre UK) suggests using >>> background Blur or a background image >>> <https://www.ncsc.gov.uk/guidance/video-conferencing-services-security-guidance-organisations> >>> for >>> staff meetings to add a degree of personal privacy. On the Web, due to >>> a lack of a standardized API for Background Blur and widespread demand, >>> developers have no options but to use ML frameworks like Tensorflow.js and >>> other WASM libraries to satisfy their customers. This Background Blur API >>> gives developers a choice to use the native platform's API. Initial >>> public proposal >>> >>> https://github.com/w3c/mediacapture-extensions/pull/49 >>> https://github.com/w3c/mediacapture-extensions/pull/61 >>> Search tags >>> >>> background blur >>> <https://chromestatus.com/features#tags:background%20blur>, webrtc >>> <https://chromestatus.com/features#tags:webrtc> >>> TAG review >>> >>> >>> >>> May be a good idea to file for a TAG review early. >>> >>> >>> >>> TAG review status >>> >>> Pending >>> Risks >>> >>> >>> Interoperability and Compatibility >>> >>> >>> >>> *Gecko*: No signal, but spec LGTM ed by Jan-Ivar (Mozilla). >>> https://github.com/mozilla/standards-positions/issues/658 >>> >>> *WebKit*: No signal but Apple co-author of explainer / PR. >>> https://lists.webkit.org/pipermail/webkit-dev/2022-June/032311.html >>> >>> *Web developers*: No signals >>> >>> *Other signals*: Almost every video-conference app has background blur >>> feature. >>> WebView application risks >>> >>> *Does this intent deprecate or change behavior of existing APIs, such >>> that it has potentially high risk for Android WebView-based applications?* >>> >>> >>> Debuggability >>> >>> >>> Is this feature fully tested by web-platform-tests >>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>> ? >>> >>> Yes ( as part of CL ) >>> Flag name >>> >>> MediaCaptureBackgroundBlur >>> Requires code in //chrome? >>> >>> False >>> Tracking bug >>> https://bugs.chromium.org/p/chromium/issues/detail?id=1338665 Estimated >>> milestones >>> >>> No milestones specified >>> >>> >>> Link to entry on the Chrome Platform Status >>> >>> https://chromestatus.com/feature/5077577782263808 >>> >>> This intent message was generated by Chrome Platform Status >>> <https://chromestatus.com/>. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191E86DD684B0D86A12B0EEF0B29%40PH0PR11MB5191.namprd11.prod.outlook.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191E86DD684B0D86A12B0EEF0B29%40PH0PR11MB5191.namprd11.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWu4kMKsspZYAchmqjLSvXKb5HT1wen0E5JqjBSyG8onw%40mail.gmail.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWu4kMKsspZYAchmqjLSvXKb5HT1wen0E5JqjBSyG8onw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPpwU5KKz%2BPuBdyTBP9w_W%3D5ny9jTErBVHpJbpYCVZmDmgLGbQ%40mail.gmail.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPpwU5KKz%2BPuBdyTBP9w_W%3D5ny9jTErBVHpJbpYCVZmDmgLGbQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F9937507ED7BDAC101FDF0B59%40PH0PR11MB5191.namprd11.prod.outlook.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F9937507ED7BDAC101FDF0B59%40PH0PR11MB5191.namprd11.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F7FB589497FC4BBED1A7F0BA9%40PH0PR11MB5191.namprd11.prod.outlook.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191F7FB589497FC4BBED1A7F0BA9%40PH0PR11MB5191.namprd11.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHcZi7RkHiy8WegxXWQG-xWtt--jwP81t4WZdT0C9PmEw%40mail.gmail.com.
