- Revision
- 278747
- Author
- [email protected]
- Date
- 2021-06-10 18:22:52 -0700 (Thu, 10 Jun 2021)
Log Message
[Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
https://bugs.webkit.org/show_bug.cgi?id=226858
rdar://77522786
Reviewed by Devin Rousso.
Source/WebCore:
Push `m_elementsWithTextRecognitionResults` from WebPage down to Page, and additionally keep track of a
TextRecognitionResult per element, as well as the size of the element at the time when we last injected text
recognition results. Use this information to tell elements to update their OCR text containers if their
dimenions have changed since the last update, after finishing a rendering update.
Test: fast/images/text-recognition/image-overlay-size-change.html
* html/HTMLElement.cpp:
(WebCore::HTMLElement::updateWithTextRecognitionResult):
Refactor this to take a `const TextRecognitionResult&` instead of a `TextRecognitionResult&&`, since this method
doesn't actually take ownership of the TextRecognitionResult. This allows us to avoid explicitly copying a
TextRecognitionResult when calling this method from `Page::updateElementsWithTextRecognitionResults`. Also, add
a CacheTextRecognitionResults enum flag as an argument, to avoid entering `Page::cacheTextRecognitionResult`
every time we update a resized element in `Page::updateElementsWithTextRecognitionResults`; since we know that
the dimensions are the only thing that changed, we simply update the cached dimensions.
Call into `Page::cacheTextRecognitionResult` to store per-element TextRecognitionResults when we
finish injecting the OCR text and data detectors into the image element.
* html/HTMLElement.h:
* page/Page.cpp:
(WebCore::Page::didCommitLoad):
Clear out both `m_textRecognitionResultsByElement` and `m_elementsWithTextRecognitionResults`.
(WebCore::Page::doAfterUpdateRendering):
At the end of each rendering update, go through `m_elementsWithTextRecognitionResults` and check if any of the
elements' dimensions have changed; if so, tell those elements to re-update their OCR text quads and data
detectors in the UA shadow tree.
(WebCore::Page::resetTextRecognitionResults):
(WebCore::Page::updateElementsWithTextRecognitionResults):
(WebCore::Page::hasCachedTextRecognitionResult const):
(WebCore::Page::cacheTextRecognitionResult):
Add the element to `m_elementsWithTextRecognitionResults` and `m_textRecognitionResultsByElement`, appending a
new entry to `m_textRecognitionResultsByElement` only if necessary.
* page/Page.h:
Source/WebKit:
Push `m_elementsWithTextRecognitionResults` from `WebKit::WebPage` down to `WebCore::Page`. See WebCore
ChangeLog for more details.
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::didCommitLoad):
(WebKit::WebPage::requestTextRecognition):
(WebKit::WebPage::updateWithTextRecognitionResult):
* WebProcess/WebPage/WebPage.h:
LayoutTests:
* fast/images/text-recognition/image-overlay-size-change-expected.txt: Added.
* fast/images/text-recognition/image-overlay-size-change.html: Added.
Modified Paths
Added Paths
Diff
Modified: trunk/LayoutTests/ChangeLog (278746 => 278747)
--- trunk/LayoutTests/ChangeLog 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/LayoutTests/ChangeLog 2021-06-11 01:22:52 UTC (rev 278747)
@@ -1,3 +1,14 @@
+2021-06-10 Wenson Hsieh <[email protected]>
+
+ [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
+ https://bugs.webkit.org/show_bug.cgi?id=226858
+ rdar://77522786
+
+ Reviewed by Devin Rousso.
+
+ * fast/images/text-recognition/image-overlay-size-change-expected.txt: Added.
+ * fast/images/text-recognition/image-overlay-size-change.html: Added.
+
2021-06-10 Arcady Goldmints-Orlov <[email protected]>
[GLIB] Unreviewed test gardening, mark some WebGL tests as passing
Added: trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change-expected.txt (0 => 278747)
--- trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change-expected.txt (rev 0)
+++ trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change-expected.txt 2021-06-11 01:22:52 UTC (rev 278747)
@@ -0,0 +1,10 @@
+PASS width is 200
+PASS height is 200
+PASS width became 50
+PASS height is 50
+PASS width became 200
+PASS height is 200
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
Added: trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change.html (0 => 278747)
--- trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change.html (rev 0)
+++ trunk/LayoutTests/fast/images/text-recognition/image-overlay-size-change.html 2021-06-11 01:22:52 UTC (rev 278747)
@@ -0,0 +1,59 @@
+<!DOCTYPE html>
+<html>
+<script src=""
+<script src=""
+<body>
+<img src=""
+<script>
+jsTestIsAsync = true;
+let image = document.querySelector("img");
+width = 0;
+height = 0;
+
+function updateImageOverlayTextDimensions() {
+ let textBoundingRect = internals.shadowRoot(image).querySelector(".image-overlay-text").getBoundingClientRect();
+ width = textBoundingRect.width;
+ height = textBoundingRect.height;
+}
+
+addEventListener("load", async () => {
+ for (let i = 0; i < 10; ++i) {
+ internals.installImageOverlay(image, [
+ {
+ topLeft : new DOMPointReadOnly(0.5, 0.5),
+ topRight : new DOMPointReadOnly(1, 0.5),
+ bottomRight : new DOMPointReadOnly(1, 1),
+ bottomLeft : new DOMPointReadOnly(0.5, 1),
+ children: [
+ {
+ text : "Hello",
+ topLeft : new DOMPointReadOnly(0.5, 0.5),
+ topRight : new DOMPointReadOnly(1, 0.5),
+ bottomRight : new DOMPointReadOnly(1, 1),
+ bottomLeft : new DOMPointReadOnly(0.5, 1),
+ }
+ ],
+ }
+ ]);
+ }
+
+ updateImageOverlayTextDimensions();
+
+ shouldBe("width", "200");
+ shouldBe("height", "200");
+
+ image.style.width = "100px";
+ setInterval(updateImageOverlayTextDimensions, 10);
+
+ await shouldBecomeEqual("width", "50");
+ shouldBe("height", "50");
+
+ image.style.width = "";
+ await shouldBecomeEqual("width", "200");
+ shouldBe("height", "200");
+
+ finishJSTest();
+});
+</script>
+</body>
+</html>
\ No newline at end of file
Modified: trunk/Source/WebCore/ChangeLog (278746 => 278747)
--- trunk/Source/WebCore/ChangeLog 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebCore/ChangeLog 2021-06-11 01:22:52 UTC (rev 278747)
@@ -1,3 +1,53 @@
+2021-06-10 Wenson Hsieh <[email protected]>
+
+ [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
+ https://bugs.webkit.org/show_bug.cgi?id=226858
+ rdar://77522786
+
+ Reviewed by Devin Rousso.
+
+ Push `m_elementsWithTextRecognitionResults` from WebPage down to Page, and additionally keep track of a
+ TextRecognitionResult per element, as well as the size of the element at the time when we last injected text
+ recognition results. Use this information to tell elements to update their OCR text containers if their
+ dimenions have changed since the last update, after finishing a rendering update.
+
+ Test: fast/images/text-recognition/image-overlay-size-change.html
+
+ * html/HTMLElement.cpp:
+ (WebCore::HTMLElement::updateWithTextRecognitionResult):
+
+ Refactor this to take a `const TextRecognitionResult&` instead of a `TextRecognitionResult&&`, since this method
+ doesn't actually take ownership of the TextRecognitionResult. This allows us to avoid explicitly copying a
+ TextRecognitionResult when calling this method from `Page::updateElementsWithTextRecognitionResults`. Also, add
+ a CacheTextRecognitionResults enum flag as an argument, to avoid entering `Page::cacheTextRecognitionResult`
+ every time we update a resized element in `Page::updateElementsWithTextRecognitionResults`; since we know that
+ the dimensions are the only thing that changed, we simply update the cached dimensions.
+
+ Call into `Page::cacheTextRecognitionResult` to store per-element TextRecognitionResults when we
+ finish injecting the OCR text and data detectors into the image element.
+
+ * html/HTMLElement.h:
+ * page/Page.cpp:
+ (WebCore::Page::didCommitLoad):
+
+ Clear out both `m_textRecognitionResultsByElement` and `m_elementsWithTextRecognitionResults`.
+
+ (WebCore::Page::doAfterUpdateRendering):
+
+ At the end of each rendering update, go through `m_elementsWithTextRecognitionResults` and check if any of the
+ elements' dimensions have changed; if so, tell those elements to re-update their OCR text quads and data
+ detectors in the UA shadow tree.
+
+ (WebCore::Page::resetTextRecognitionResults):
+ (WebCore::Page::updateElementsWithTextRecognitionResults):
+ (WebCore::Page::hasCachedTextRecognitionResult const):
+ (WebCore::Page::cacheTextRecognitionResult):
+
+ Add the element to `m_elementsWithTextRecognitionResults` and `m_textRecognitionResultsByElement`, appending a
+ new entry to `m_textRecognitionResultsByElement` only if necessary.
+
+ * page/Page.h:
+
2021-06-10 Chris Dumez <[email protected]>
REGRESSION: (r278544) [ Mac-wk1 ] media/media-continues-playing-after-replace-source.html is a flaky failure
Modified: trunk/Source/WebCore/html/HTMLElement.cpp (278746 => 278747)
--- trunk/Source/WebCore/html/HTMLElement.cpp 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebCore/html/HTMLElement.cpp 2021-06-11 01:22:52 UTC (rev 278747)
@@ -1331,7 +1331,7 @@
#if ENABLE(IMAGE_ANALYSIS)
-void HTMLElement::updateWithTextRecognitionResult(TextRecognitionResult&& result)
+void HTMLElement::updateWithTextRecognitionResult(const TextRecognitionResult& result, CacheTextRecognitionResults cacheTextRecognitionResults)
{
RefPtr<HTMLDivElement> previousContainer;
if (auto shadowRoot = userAgentShadowRoot(); shadowRoot && hasImageOverlay()) {
@@ -1509,6 +1509,11 @@
if (auto frame = makeRefPtr(document().frame()))
frame->eventHandler().scheduleCursorUpdate();
+
+ if (cacheTextRecognitionResults == CacheTextRecognitionResults::Yes) {
+ if (auto* page = document().page())
+ page->cacheTextRecognitionResult(*this, containerSize, result);
+ }
}
#endif // ENABLE(IMAGE_ANALYSIS)
Modified: trunk/Source/WebCore/html/HTMLElement.h (278746 => 278747)
--- trunk/Source/WebCore/html/HTMLElement.h 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebCore/html/HTMLElement.h 2021-06-11 01:22:52 UTC (rev 278747)
@@ -139,7 +139,8 @@
WEBCORE_EXPORT static bool isImageOverlayText(const Node*);
#if ENABLE(IMAGE_ANALYSIS)
- WEBCORE_EXPORT void updateWithTextRecognitionResult(TextRecognitionResult&&);
+ enum class CacheTextRecognitionResults : bool { No, Yes };
+ WEBCORE_EXPORT void updateWithTextRecognitionResult(const TextRecognitionResult&, CacheTextRecognitionResults = CacheTextRecognitionResults::Yes);
#endif
#if PLATFORM(IOS_FAMILY)
Modified: trunk/Source/WebCore/page/Page.cpp (278746 => 278747)
--- trunk/Source/WebCore/page/Page.cpp 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebCore/page/Page.cpp 2021-06-11 01:22:52 UTC (rev 278747)
@@ -136,6 +136,7 @@
#include "StyleScope.h"
#include "SubframeLoader.h"
#include "TextIterator.h"
+#include "TextRecognitionResult.h"
#include "TextResourceDecoder.h"
#include "UserContentProvider.h"
#include "UserContentURLPattern.h"
@@ -1274,6 +1275,10 @@
resetSeenPlugins();
resetSeenMediaEngines();
+
+#if ENABLE(IMAGE_ANALYSIS)
+ resetTextRecognitionResults();
+#endif
}
void Page::didFinishLoad()
@@ -1661,6 +1666,10 @@
});
#endif
+#if ENABLE(IMAGE_ANALYSIS)
+ updateElementsWithTextRecognitionResults();
+#endif
+
prioritizeVisibleResources();
m_renderingUpdateRemainingSteps.last().remove(RenderingUpdateStep::EventRegionUpdate);
@@ -3603,4 +3612,61 @@
return ts;
}
+#if ENABLE(IMAGE_ANALYSIS)
+
+void Page::updateElementsWithTextRecognitionResults()
+{
+ if (m_textRecognitionResultsByElement.isEmpty())
+ return;
+
+ m_textRecognitionResultsByElement.removeAllMatching([](auto& elementAndResult) {
+ return !elementAndResult.first;
+ });
+
+ for (auto& [element, resultAndSize] : m_textRecognitionResultsByElement) {
+ if (!element || !element->isConnected())
+ continue;
+
+ auto protectedElement = makeRef(*element);
+ auto& [result, offsetSize] = resultAndSize;
+ IntSize newOffsetSize { protectedElement->offsetWidth(), protectedElement->offsetHeight() };
+ if (offsetSize == newOffsetSize)
+ continue;
+
+ offsetSize = newOffsetSize;
+ protectedElement->updateWithTextRecognitionResult(result, HTMLElement::CacheTextRecognitionResults::No);
+ }
+}
+
+bool Page::hasCachedTextRecognitionResult(const HTMLElement& element) const
+{
+ return m_elementsWithTextRecognitionResults.contains(element);
+}
+
+void Page::cacheTextRecognitionResult(const HTMLElement& element, const IntSize& offsetSize, const TextRecognitionResult& result)
+{
+ m_elementsWithTextRecognitionResults.add(element);
+
+ auto index = m_textRecognitionResultsByElement.findMatching([&](auto& elementAndResult) {
+ return elementAndResult.first == &element;
+ });
+
+ if (index == notFound)
+ m_textRecognitionResultsByElement.append({ makeWeakPtr(element), { result, offsetSize } });
+ else
+ m_textRecognitionResultsByElement[index].second = { result, offsetSize };
+
+ m_textRecognitionResultsByElement.removeAllMatching([](auto& elementAndResult) {
+ return !elementAndResult.first;
+ });
+}
+
+void Page::resetTextRecognitionResults()
+{
+ m_textRecognitionResultsByElement.clear();
+ m_elementsWithTextRecognitionResults.clear();
+}
+
+#endif // ENABLE(IMAGE_ANALYSIS)
+
} // namespace WebCore
Modified: trunk/Source/WebCore/page/Page.h (278746 => 278747)
--- trunk/Source/WebCore/page/Page.h 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebCore/page/Page.h 2021-06-11 01:22:52 UTC (rev 278747)
@@ -107,11 +107,13 @@
class FocusController;
class FormData;
class Frame;
+class HTMLElement;
class HTMLMediaElement;
class HistoryItem;
class ImageOverlayController;
class InspectorClient;
class InspectorController;
+class IntSize;
class LibWebRTCProvider;
class LowPowerModeNotifier;
class MediaCanStartListener;
@@ -158,6 +160,7 @@
class WheelEventTestMonitor;
struct SimpleRange;
+struct TextRecognitionResult;
using PlatformDisplayID = uint32_t;
using SharedStringHash = uint32_t;
@@ -860,6 +863,11 @@
LoadSchedulingMode loadSchedulingMode() const { return m_loadSchedulingMode; }
void setLoadSchedulingMode(LoadSchedulingMode);
+#if ENABLE(IMAGE_ANALYSIS)
+ WEBCORE_EXPORT bool hasCachedTextRecognitionResult(const HTMLElement&) const;
+ void cacheTextRecognitionResult(const HTMLElement&, const IntSize& offsetSize, const TextRecognitionResult&);
+#endif
+
private:
struct Navigation {
RegistrableDomain domain;
@@ -905,6 +913,11 @@
WheelEventTestMonitor& ensureWheelEventTestMonitor();
+#if ENABLE(IMAGE_ANALYSIS)
+ void resetTextRecognitionResults();
+ void updateElementsWithTextRecognitionResults();
+#endif
+
const std::unique_ptr<Chrome> m_chrome;
const std::unique_ptr<DragCaretController> m_dragCaretController;
@@ -1175,6 +1188,12 @@
const bool m_httpsUpgradeEnabled { true };
mutable MediaSessionGroupIdentifier m_mediaSessionGroupIdentifier;
+
+#if ENABLE(IMAGE_ANALYSIS)
+ // FIXME: These should be refactored to use a weak hash map of HTMLElement to std::pair<TextRecognitionResult, IntSize>.
+ Vector<std::pair<WeakPtr<HTMLElement>, std::pair<TextRecognitionResult, IntSize>>> m_textRecognitionResultsByElement;
+ WeakHashSet<HTMLElement> m_elementsWithTextRecognitionResults;
+#endif
};
inline PageGroup& Page::group()
Modified: trunk/Source/WebKit/ChangeLog (278746 => 278747)
--- trunk/Source/WebKit/ChangeLog 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebKit/ChangeLog 2021-06-11 01:22:52 UTC (rev 278747)
@@ -1,3 +1,20 @@
+2021-06-10 Wenson Hsieh <[email protected]>
+
+ [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
+ https://bugs.webkit.org/show_bug.cgi?id=226858
+ rdar://77522786
+
+ Reviewed by Devin Rousso.
+
+ Push `m_elementsWithTextRecognitionResults` from `WebKit::WebPage` down to `WebCore::Page`. See WebCore
+ ChangeLog for more details.
+
+ * WebProcess/WebPage/WebPage.cpp:
+ (WebKit::WebPage::didCommitLoad):
+ (WebKit::WebPage::requestTextRecognition):
+ (WebKit::WebPage::updateWithTextRecognitionResult):
+ * WebProcess/WebPage/WebPage.h:
+
2021-06-10 Per Arne Vollan <[email protected]>
[iOS] Sync Accessibility preferences
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp (278746 => 278747)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2021-06-11 01:22:52 UTC (rev 278747)
@@ -6290,7 +6290,6 @@
completionHandler({ });
}
m_elementsPendingTextRecognition.clear();
- m_elementsWithTextRecognitionResults.clear();
#endif
#if ENABLE(RESOURCE_LOAD_STATISTICS)
@@ -7396,11 +7395,11 @@
return;
}
- if (m_elementsWithTextRecognitionResults.contains(element)) {
+ auto htmlElement = makeRef(downcast<HTMLElement>(element));
+ if (corePage()->hasCachedTextRecognitionResult(htmlElement.get())) {
if (completion) {
- ASSERT(is<HTMLElement>(element));
RefPtr<Element> imageOverlayHost;
- if (is<HTMLElement>(element) && downcast<HTMLElement>(element).hasImageOverlay())
+ if (htmlElement->hasImageOverlay())
imageOverlayHost = makeRefPtr(element);
completion(WTFMove(imageOverlayHost));
}
@@ -7466,8 +7465,7 @@
return;
auto& htmlElement = downcast<HTMLElement>(*protectedElement);
- htmlElement.updateWithTextRecognitionResult(WTFMove(result));
- protectedPage->m_elementsWithTextRecognitionResults.add(htmlElement);
+ htmlElement.updateWithTextRecognitionResult(result);
auto matchIndex = protectedPage->m_elementsPendingTextRecognition.findMatching([&] (auto& elementAndCompletionHandlers) {
return elementAndCompletionHandlers.first == &htmlElement;
@@ -7484,7 +7482,7 @@
});
}
-void WebPage::updateWithTextRecognitionResult(TextRecognitionResult&& result, const ElementContext& context, const FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&& completionHandler)
+void WebPage::updateWithTextRecognitionResult(const TextRecognitionResult& result, const ElementContext& context, const FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&& completionHandler)
{
auto elementToUpdate = elementForContext(context);
if (!is<HTMLElement>(elementToUpdate)) {
@@ -7492,7 +7490,7 @@
return;
}
- downcast<HTMLElement>(*elementToUpdate).updateWithTextRecognitionResult(WTFMove(result));
+ downcast<HTMLElement>(*elementToUpdate).updateWithTextRecognitionResult(result);
auto hitTestResult = corePage()->mainFrame().eventHandler().hitTestResultAtPoint(roundedIntPoint(location), {
HitTestRequest::Type::ReadOnly,
HitTestRequest::Type::Active,
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.h (278746 => 278747)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.h 2021-06-11 01:02:06 UTC (rev 278746)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.h 2021-06-11 01:22:52 UTC (rev 278747)
@@ -244,6 +244,7 @@
struct RequestStorageAccessResult;
struct RunJavaScriptParameters;
struct TextCheckingResult;
+struct TextRecognitionResult;
struct ViewportArguments;
#if ENABLE(ATTACHMENT_ELEMENT)
@@ -1404,7 +1405,7 @@
#if ENABLE(IMAGE_ANALYSIS)
void requestTextRecognition(WebCore::Element&, CompletionHandler<void(RefPtr<WebCore::Element>&&)>&&);
- void updateWithTextRecognitionResult(WebCore::TextRecognitionResult&&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
+ void updateWithTextRecognitionResult(const WebCore::TextRecognitionResult&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
#endif
#if HAVE(TRANSLATION_UI_SERVICES) && ENABLE(CONTEXT_MENUS)
@@ -2348,7 +2349,6 @@
#if ENABLE(IMAGE_ANALYSIS)
Vector<std::pair<WeakPtr<WebCore::HTMLElement>, Vector<CompletionHandler<void(RefPtr<WebCore::Element>&&)>>>> m_elementsPendingTextRecognition;
- WeakHashSet<WebCore::HTMLElement> m_elementsWithTextRecognitionResults;
#endif
#if ENABLE(WEBXR) && PLATFORM(COCOA)
