Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: f9ad94e23bc21d20b48be8c4304e29215b7c9023
https://github.com/WebKit/WebKit/commit/f9ad94e23bc21d20b48be8c4304e29215b7c9023
Author: Wenson Hsieh <[email protected]>
Date: 2026-06-28 (Sun, 28 Jun 2026)
Changed paths:
M
LayoutTests/fast/text-extraction/debug-text-extraction-interactions-expected.txt
M LayoutTests/fast/text-extraction/debug-text-extraction-interactions.html
M Source/WebCore/page/text-extraction/TextExtraction.cpp
M Source/WebKit/Shared/TextExtractionToStringConversion.cpp
M Source/WebKit/Shared/TextExtractionToStringConversion.h
M Source/WebKit/Sources.txt
M Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm
A Source/WebKit/UIProcess/TextExtractionCache.cpp
A Source/WebKit/UIProcess/TextExtractionCache.h
M Source/WebKit/UIProcess/WebPageProxy.cpp
M Source/WebKit/UIProcess/WebPageProxy.h
M Source/WebKit/UIProcess/WebPageProxyInternals.h
M Source/WebKit/WebKit.xcodeproj/project.pbxproj
Log Message:
-----------
[Text Extraction] Work towards making it possible for clients to target stale
node UIDs
https://bugs.webkit.org/show_bug.cgi?id=318038
rdar://180823125
Reviewed by Richard Robinson.
It's possible for agentic text extraction clients to issue clicks over stale
unique node `uid`s; for
instance, consider an agent that navigates from one page:
```
…
button uid=2845 haspopup=true label=Menu
button uid=2846 haspopup=true
image uid=2847 src=avatar.svg alt='Account menu'
'MyChart'
'by Epic'
…
```
…to another page:
```
…
button uid=3004 haspopup=true label=Menu
button uid=3005 haspopup=true
image uid=3006 src=avatar.svg alt='Account menu'
'MyChart'
'by Epic'
…
```
…and then attempts to issue a click on `uid=2845`. We'll currently reject the
`click` interaction,
communicating in the summary string that the targeted node UID is not rendered.
This can end up
confusing or otherwise misaligning a weaker model, which may end up assuming
that (in this
particular example) the `Menu` button is simply no longer accessible in the
current page, rather
than realizing that it had provided a stale `uid`.
To mitigate this, we work towards a mechanism where:
- We cache the last few text extraction results.
- When a `uid` target is stale, we fall back to searching for that `uid` in
these cached results.
- If the context before or after the `uid` is identical (ignoring the `uid`s
themselves) to the
context before or after a `uid` in the latest text extraction results taken
from the current
page, then treat it as if the agent had targeted the new, matching `uid`
instead.
This patch only introduces the concept of an text extraction result cache,
which we'll later consult
to resolve stale `uid`s to active `uid`s in the current context, as a fallback.
* Source/WebCore/page/text-extraction/TextExtraction.cpp:
(WebCore::TextExtraction::invalidNodeIdentifierDescription):
(WebCore::TextExtraction::resolveMouseTarget):
* Source/WebKit/Shared/TextExtractionToStringConversion.cpp:
(WebKit::TextExtractionAggregator::~TextExtractionAggregator):
(WebKit::TextExtractionAggregator::takeResults):
(WebKit::TextExtractionAggregator::addResult):
(WebKit::lineWithoutNodeIdentifier):
(WebKit::addPartsForItem):
While extracting text, we now also build up a parallel representation of the
text extraction
results, where the `uid` is separated out from the rest of the content. This
representation allows
us to later efficiently:
1. Check which `uid`s are present in a cached text extraction result.
2. If present, we can easily check the surrounding context around the `uid` to
see if it matches.
* Source/WebKit/Shared/TextExtractionToStringConversion.h:
* Source/WebKit/Sources.txt:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView
_extractDebugTextWithConfigurationWithoutUpdatingFilterRules:assertionScope:completionHandler:]):
* Source/WebKit/UIProcess/TextExtractionCache.cpp: Added.
(WebKit::TextExtractionCache::add):
(WebKit::TextExtractionCache::contextWindowBefore):
(WebKit::TextExtractionCache::contextWindowAfter):
(WebKit::TextExtractionCache::resolve const):
* Source/WebKit/UIProcess/TextExtractionCache.h: Added.
(WebKit::TextExtractionCache::clear):
Add a helper class that represents a cached text extraction result, which we
use to do the fallback
`uid` lookup above.
* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::textExtractionCache):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/WebPageProxyInternals.h:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
Canonical link: https://commits.webkit.org/315981@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
