Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 1207e682053fff75cd104d5ae8a02f138c50f72a
https://github.com/WebKit/WebKit/commit/1207e682053fff75cd104d5ae8a02f138c50f72a
Author: Wenson Hsieh <[email protected]>
Date: 2026-09-04 (Fri, 04 Sep 2026)
Changed paths:
M Source/WebCore/page/text-extraction/TextExtraction.cpp
M Source/WebCore/page/text-extraction/TextExtractionTypes.h
M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
M Source/WebKit/UIProcess/API/Cocoa/WKWebView+TextExtraction.mm
M Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm
M Source/WebKit/UIProcess/API/Cocoa/WKWebViewInternal.h
M Source/WebKit/UIProcess/TextExtractionCache.cpp
M Source/WebKit/UIProcess/TextExtractionCache.h
M Source/WebKit/UIProcess/WebFrameProxy.cpp
M Source/WebKit/UIProcess/WebFrameProxy.h
M Source/WebKit/WebProcess/WebPage/WebFrame.cpp
M Source/WebKit/WebProcess/WebPage/WebFrame.h
M Source/WebKit/WebProcess/WebPage/WebFrame.messages.in
M Tools/TestWebKitAPI/Tests/WebKit/WKWebView/TextExtractionTests.mm
Log Message:
-----------
[Text Extraction] Stale node UID resolution heuristic should surface failure
in the case where the resolved node is stale
https://bugs.webkit.org/show_bug.cgi?id=323397
rdar://186483543
Reviewed by Abrar Rahman Protyasha.
The stale node resolution heuristic (for Safari MCP, Automatically Fix
Passwords, etc.) currently
allows agents to interact with stale node `uid`s, as long as the stale node
`uid` can be mapped to a
node in the DOM that appeared in a later extraction with matching
self-and-surrounding context. This
is achieved by maintaining a sliding window of recent text extraction results
on the web view, such
that we're able to both:
1. Detect when the agent tries to interact with a stale `uid` that no longer
corresponds to
anything in the current page.
2. Check if anything in the later text extraction looks (nearly) identical to
the old context
(ignoring differences in `uid` and some other transient attributes), and
instead of failing,
handle the interaction as if the agent tried to interact with that newer
node `uid`.
This is especially crucial to keep weaker models from consistently failing to
issue valid tool calls
when reasoning about long-horizon tasks, if it tries to interact with stale
nodes. However, this
mechanism currently doesn't check that this newer resolved node isn't (itself)
stale; as a result,
the interaction result and description can end up confusing agents even further
("node uid `X` was
stale, so we resolved it to `Y`", followed by "Failed to interact with `Y`
because `Y` was stale").
To fix this, we add one final sanity check to ensure that the node we end up
resolving the stale uid
to isn't also stale. `resolve()` now returns the remapping candidates in order
of preference, and we
ask the web process for the first one that's still connected before committing
to it. If none of
them are, we report the original failure as a stale uid and tell the agent to
re-extract the page
rather than interacting with a node we already know is gone.
Test:
TextExtractionTests.InteractionDebugDescriptionWithUnresolvableNodeIdentifier
TextExtractionTests.InteractionDebugDescriptionWithoutTargetElement
TextExtractionTests.InteractionReportsStaleNodeWhenRemapCandidateIsAlsoStale
* Source/WebCore/page/text-extraction/TextExtraction.cpp:
(WebCore::TextExtraction::interactionDescription):
* Source/WebCore/page/text-extraction/TextExtractionTypes.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView+TextExtraction.mm:
(noteForRemappedStaleNode):
(findFirstConnectedRemapCandidate):
(-[WKWebView
_performInteraction:inFrame:actionType:staleNodeResolution:completionHandler:]):
(-[WKWebView
_retryInteractionWithConnectedRemapCandidate:actionType:requestedNodeIdentifier:remapCandidates:failureDescription:completionHandler:]):
(-[WKWebView _describeInteraction:completionHandler:]):
(-[WKWebView
_describeInteraction:inFrame:staleNodeResolution:completionHandler:]):
(-[WKWebView
_retryDescribingInteractionWithConnectedRemapCandidate:requestedNodeIdentifier:remapCandidates:completionHandler:]):
(-[WKWebView
_performInteraction:inFrame:actionType:nodeIdentifier:staleNodeNote:shouldResolveStaleNodeIdentifier:completionHandler:]):
Deleted.
(-[WKWebView
_describeInteraction:inFrame:nodeIdentifier:staleNodeNote:shouldResolveStaleNodeIdentifier:completionHandler:]):
Deleted.
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _performInteraction:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewInternal.h:
* Source/WebKit/UIProcess/TextExtractionCache.cpp:
(WebKit::TextExtractionCache::resolve const):
* Source/WebKit/UIProcess/TextExtractionCache.h:
(WebKit::StaleNodeResolutionState::didRemap const):
* Source/WebKit/UIProcess/WebFrameProxy.cpp:
(WebKit::WebFrameProxy::findFirstConnectedNode):
* Source/WebKit/UIProcess/WebFrameProxy.h:
* Source/WebKit/WebProcess/WebPage/WebFrame.cpp:
(WebKit::WebFrame::describeTextExtractionInteraction):
(WebKit::WebFrame::findFirstConnectedNode):
* Source/WebKit/WebProcess/WebPage/WebFrame.h:
* Source/WebKit/WebProcess/WebPage/WebFrame.messages.in:
* Tools/TestWebKitAPI/Tests/WebKit/WKWebView/TextExtractionTests.mm:
(TestWebKitAPI::TEST(TextExtractionTests,
InteractionDebugDescriptionWithStaleNodeIdentifier)):
(TestWebKitAPI::TEST(TextExtractionTests,
InteractionDebugDescriptionWithoutTargetElement)):
(TestWebKitAPI::TEST(TextExtractionTests,
InteractionReportsStaleNodeWhenRemapCandidateIsAlsoStale)):
Canonical link: https://commits.webkit.org/320523@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
