Thank you very much Philip, About the concern that a repaint based approach (similar to NodeImage) will not have the backgrounds or other higher level effects, can we say that on the other hand (specially since this function is explicitly querying the image for a certain node) the full-page-paint+crop approach also suffers from the possibility that the node maybe be partially covered by another node, or being out of the screen's visible area (needing scroll), or even being covered by a dialog box?
On Mon, Jan 10, 2022 at 8:29 PM Philip Rogers <[email protected]> wrote: > > > On Mon, Jan 10, 2022 at 6:51 AM Ramin Halavati <[email protected]> > wrote: > >> Thank you Philip. >> >> Since this accessibility support function is called for all renderer >> types (in ChromeOS including Arc++ and Ash), there would be a better >> consistency if we have the snapshotting code in the renderer. Therefore I >> am trying to avoid using the browser side ui::GrabViewSnapshot (as in >> DevTools), unless the alternative approach proves to be too complicated or >> unnecessarily heavy. >> >> I tried using DataTransfer::NodeImage in this draft CL >> <http://crrev.com/c/3370381>, but my understanding is that since it >> triggers a repaint, we should make sure it is not in the middle of a >> lifecycle update to avoid conflicts. Is there a notification of a >> signal that tells when it's safe to request it? >> > > It is important that we do not introduce lifecycle violations but I think > that may already be guaranteed from when the accessibility code runs. For > example, AxObject::GetRelativeBounds cannot be called from within a > lifecycle update either. You can probably just synchronously call the paint > function you need. > > >> I assume it's the same for the SpoolSinglePage as it also redraws the >> image. And performance wise, can we say that as this approach triggers a >> repaint, it's heavier than the browser based snapshotting (using >> ui::GrabViewSnapshot), as the latter is using the already painted output >> and does not require a repaint? >> > > The paint code in NodeImage is really just a walk of layout datastructures > and isn't more expensive than things like layout, style, etc (i.e., it's > O(|layout|)). Doing this for each object would be wildly inefficient though. > > Expanding on the thing about this being an ambiguous problem: > data:text/html,<body style="background: rebeccapurple; filter: > blur(1px)"><div style="height: 100px; width: 100px; border: 1px solid > white; color:white;" draggable=true>hello world</div> > The drag image for the div will only contain white and transparent pixels, > and will not be blurred. If you want the purple background too, you'll want > to do the full-page-paint+crop approach. > > >> >> Best, >> Ramin >> >> On Fri, Jan 7, 2022 at 11:06 PM Philip Rogers <[email protected]> wrote: >> >>> This is a deceptively difficult problem in general. Our paint system >>> uses the CSS definition of paint order which starts painting at stacking >>> contexts and it is challenging to exclude non-stacked siblings of a >>> non-stacked target element. Some other complications are whether ancestor >>> effects/clips should be applied, and whether content behind the target >>> should be included. >>> >>> DataTransfer::NodeImage is an attempt at this. Another option is to get >>> the entire page painted (examples: devtools uses >>> PageHandler::CaptureScreenshot and printing uses SpoolSinglePage), then >>> crop it to the target element. >>> >>> On Fri, Jan 7, 2022 at 7:01 AM 'Ramin Halavati' via blink-dev < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I am trying to find a way to get pixels for any general node in >>>> *third_party/blink/renderer/modules/accessibility. *We already have >>>> AXNodeObject::ImageDataUrl >>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/accessibility/ax_node_object.cc;drc=a3ae56ec162fcbf9ee400e9ff80f1ae572957fd3;bpv=1;bpt=1;l=2355?gsn=ImageDataUrl&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.cc%23rau1_dUeeJHt__0C5nqtyH5ouOMnhdEN55IRAHfVRMk&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.h%23z0USMj0KE_TnmxgTOdb4D-K1NmRUt1i1gyvtbSdYs5w> >>>> function >>>> to return the bitmap of a certain node, but it is only supporting HTML >>>> element types that already include a picture (namely image, canvas, video) >>>> and I am trying to expand to to any general element type. >>>> >>>> So far I've been looking for a way to get access to a painter or >>>> compositor, but I did not find any promising solution yet. Can you point me >>>> in the right direction? Is it even expected at this abstraction level? >>>> >>>> Thanks, >>>> Ramin >>>> >>>> P.S., Sorry for naive questions and please point me to the right email >>>> thread if this is not. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "blink-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com >>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQh_h-56JQD6XDHRcMms89NP2N1zhsGodZeawBLGuiRLWA%40mail.gmail.com.
