- Revision
- 292205
- Author
- [email protected]
- Date
- 2022-03-31 22:52:48 -0700 (Thu, 31 Mar 2022)
Log Message
Add a heuristic to identify and extract the prominent video element in element fullscreen
https://bugs.webkit.org/show_bug.cgi?id=238579
Reviewed by Jer Noble.
Source/WebCore:
Mark a couple of methods as WEBCORE_EXPORT so that they can be invoked from the WebKit layer.
No change in behavior (yet).
* dom/Element.h:
* html/HTMLMediaElement.h:
Drive-by-fix: also remove an (unused) function declaration with no implementation
(`updatePlaybackControlsManager()`).
Source/WebKit:
Add a heuristic to WebFullScreenManager that identifies and keeps track of the largest, rendered video element
in element fullscreen mode (that is, the video element with the largest area that's inside of the subtree of the
fullscreen container element). We couple this with a mechanism to detect when such a video has been paused for a
short (250 ms) delay.
These mechanisms will be used in a future patch to support "video extraction" in element fullscreen. See below
for more details.
* WebProcess/FullScreen/WebFullScreenManager.cpp:
Turn WebFullScreenManager into an EventListener subclass, so that we can observe the "playing", "pause", and
"seeking" events on the main video element (if it exists). See `handleEvent` below for more information.
(WebKit::WebFullScreenManager::WebFullScreenManager):
(WebKit::WebFullScreenManager::exitFullScreenForElement):
Clear out the main video element upon exiting fullscreen.
(WebKit::WebFullScreenManager::didEnterFullScreen):
After we've finished entering element fullsreen, traverse the DOM in search of the largest video element (area-
wise). If the container for element fullscreen is, itself, a video element, then automatically treat that
element as the main video.
(WebKit::WebFullScreenManager::mainVideoElementExtractionTimerFired):
(WebKit::WebFullScreenManager::scheduleMainVideoElementExtraction):
(WebKit::WebFullScreenManager::endMainVideoElementExtractionIfNeeded):
For now, these are essentially no-ops; add FIXME's to call into WebPage to begin or cancel video extraction.
These will be implemented in an upcoming patch, that connects this element fullscreen code to a more generic
fullscreen video extraction mechanism implemented entirely in the client layer.
(WebKit::WebFullScreenManager::setMainVideoElement):
Add a helper method to set the main video element, adding or removing event listeners as needed.
(WebKit::WebFullScreenManager::handleEvent):
When observing any of the "playing", "pause", and "seeking" events, update the main video extraction timer;
that is, if the main video element is paused, restart the timer; otherwise, stop it entirely.
* WebProcess/FullScreen/WebFullScreenManager.h:
Modified Paths
Diff
Modified: trunk/Source/WebCore/ChangeLog (292204 => 292205)
--- trunk/Source/WebCore/ChangeLog 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebCore/ChangeLog 2022-04-01 05:52:48 UTC (rev 292205)
@@ -1,3 +1,19 @@
+2022-03-31 Wenson Hsieh <[email protected]>
+
+ Add a heuristic to identify and extract the prominent video element in element fullscreen
+ https://bugs.webkit.org/show_bug.cgi?id=238579
+
+ Reviewed by Jer Noble.
+
+ Mark a couple of methods as WEBCORE_EXPORT so that they can be invoked from the WebKit layer.
+ No change in behavior (yet).
+
+ * dom/Element.h:
+ * html/HTMLMediaElement.h:
+
+ Drive-by-fix: also remove an (unused) function declaration with no implementation
+ (`updatePlaybackControlsManager()`).
+
2022-03-31 Said Abou-Hallawa <[email protected]>
[GPU Process] [iOS] Vertical text is incorrectly displaced
Modified: trunk/Source/WebCore/dom/Element.h (292204 => 292205)
--- trunk/Source/WebCore/dom/Element.h 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebCore/dom/Element.h 2022-04-01 05:52:48 UTC (rev 292205)
@@ -214,7 +214,7 @@
// This does not update layout, and uses absoluteBoundingBoxRect().
WEBCORE_EXPORT IntRect boundingBoxInRootViewCoordinates() const;
- std::optional<std::pair<RenderObject*, FloatRect>> boundingAbsoluteRectWithoutLayout() const;
+ WEBCORE_EXPORT std::optional<std::pair<RenderObject*, FloatRect>> boundingAbsoluteRectWithoutLayout() const;
WEBCORE_EXPORT FloatRect boundingClientRect();
Modified: trunk/Source/WebCore/html/HTMLMediaElement.h (292204 => 292205)
--- trunk/Source/WebCore/html/HTMLMediaElement.h 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebCore/html/HTMLMediaElement.h 2022-04-01 05:52:48 UTC (rev 292205)
@@ -423,8 +423,8 @@
void textTrackReadyStateChanged(TextTrack*);
void updateTextTrackRepresentationImageIfNeeded();
- bool addEventListener(const AtomString& eventType, Ref<EventListener>&&, const AddEventListenerOptions&) override;
- bool removeEventListener(const AtomString& eventType, EventListener&, const EventListenerOptions&) override;
+ WEBCORE_EXPORT bool addEventListener(const AtomString& eventType, Ref<EventListener>&&, const AddEventListenerOptions&) override;
+ WEBCORE_EXPORT bool removeEventListener(const AtomString& eventType, EventListener&, const EventListenerOptions&) override;
#if ENABLE(WIRELESS_PLAYBACK_TARGET)
void webkitShowPlaybackTargetPicker();
@@ -955,7 +955,6 @@
void updateShouldAutoplay();
void pauseAfterDetachedTask();
- void updatePlaybackControlsManager();
void schedulePlaybackControlsManagerUpdate();
void playbackControlsManagerBehaviorRestrictionsTimerFired();
Modified: trunk/Source/WebKit/ChangeLog (292204 => 292205)
--- trunk/Source/WebKit/ChangeLog 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebKit/ChangeLog 2022-04-01 05:52:48 UTC (rev 292205)
@@ -1,3 +1,53 @@
+2022-03-31 Wenson Hsieh <[email protected]>
+
+ Add a heuristic to identify and extract the prominent video element in element fullscreen
+ https://bugs.webkit.org/show_bug.cgi?id=238579
+
+ Reviewed by Jer Noble.
+
+ Add a heuristic to WebFullScreenManager that identifies and keeps track of the largest, rendered video element
+ in element fullscreen mode (that is, the video element with the largest area that's inside of the subtree of the
+ fullscreen container element). We couple this with a mechanism to detect when such a video has been paused for a
+ short (250 ms) delay.
+
+ These mechanisms will be used in a future patch to support "video extraction" in element fullscreen. See below
+ for more details.
+
+ * WebProcess/FullScreen/WebFullScreenManager.cpp:
+
+ Turn WebFullScreenManager into an EventListener subclass, so that we can observe the "playing", "pause", and
+ "seeking" events on the main video element (if it exists). See `handleEvent` below for more information.
+
+ (WebKit::WebFullScreenManager::WebFullScreenManager):
+ (WebKit::WebFullScreenManager::exitFullScreenForElement):
+
+ Clear out the main video element upon exiting fullscreen.
+
+ (WebKit::WebFullScreenManager::didEnterFullScreen):
+
+ After we've finished entering element fullsreen, traverse the DOM in search of the largest video element (area-
+ wise). If the container for element fullscreen is, itself, a video element, then automatically treat that
+ element as the main video.
+
+ (WebKit::WebFullScreenManager::mainVideoElementExtractionTimerFired):
+ (WebKit::WebFullScreenManager::scheduleMainVideoElementExtraction):
+ (WebKit::WebFullScreenManager::endMainVideoElementExtractionIfNeeded):
+
+ For now, these are essentially no-ops; add FIXME's to call into WebPage to begin or cancel video extraction.
+ These will be implemented in an upcoming patch, that connects this element fullscreen code to a more generic
+ fullscreen video extraction mechanism implemented entirely in the client layer.
+
+ (WebKit::WebFullScreenManager::setMainVideoElement):
+
+ Add a helper method to set the main video element, adding or removing event listeners as needed.
+
+ (WebKit::WebFullScreenManager::handleEvent):
+
+ When observing any of the "playing", "pause", and "seeking" events, update the main video extraction timer;
+ that is, if the main video element is paused, restart the timer; otherwise, stop it entirely.
+
+ * WebProcess/FullScreen/WebFullScreenManager.h:
+
2022-03-31 Chris Dumez <[email protected]>
Prepare WebKit/ & WebKitLegacy/ for making the String(const char*) constructor explicit
Modified: trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.cpp (292204 => 292205)
--- trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.cpp 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.cpp 2022-04-01 05:52:48 UTC (rev 292205)
@@ -33,7 +33,9 @@
#include "WebFrame.h"
#include "WebFullScreenManagerProxyMessages.h"
#include "WebPage.h"
+#include <WebCore/AddEventListenerOptions.h>
#include <WebCore/Color.h>
+#include <WebCore/EventNames.h>
#include <WebCore/Frame.h>
#include <WebCore/FrameView.h>
#include <WebCore/FullscreenManager.h>
@@ -42,6 +44,7 @@
#include <WebCore/RenderLayerBacking.h>
#include <WebCore/RenderView.h>
#include <WebCore/Settings.h>
+#include <WebCore/TypedElementDescendantIterator.h>
#include <WebCore/UserGestureIndicator.h>
#if PLATFORM(IOS_FAMILY) || (PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE))
@@ -71,7 +74,11 @@
}
WebFullScreenManager::WebFullScreenManager(WebPage* page)
- : m_page(page)
+ : WebCore::EventListener(WebCore::EventListener::CPPEventListenerType)
+ , m_page(page)
+#if ENABLE(VIDEO)
+ , m_mainVideoElementExtractionTimer(RunLoop::main(), this, &WebFullScreenManager::mainVideoElementExtractionTimerFired)
+#endif
{
}
@@ -152,6 +159,9 @@
{
LOG(Fullscreen, "WebFullScreenManager %p exitFullScreenForElement(%p) - fullscreen element %p", this, element, m_element.get());
m_page->injectedBundleFullScreenClient().exitFullScreenForElement(m_page.get(), element);
+#if ENABLE(VIDEO)
+ setMainVideoElement(nullptr);
+#endif
}
void WebFullScreenManager::willEnterFullScreen()
@@ -191,6 +201,35 @@
auto* currentPlaybackControlsElement = m_page->playbackSessionManager().currentPlaybackControlsElement();
setPIPStandbyElement(dynamicDowncast<HTMLVideoElement>(currentPlaybackControlsElement));
#endif
+
+#if ENABLE(VIDEO)
+ setMainVideoElement([&]() -> RefPtr<WebCore::HTMLVideoElement> {
+ if (auto video = dynamicDowncast<WebCore::HTMLVideoElement>(*m_element))
+ return video;
+
+ RefPtr<WebCore::HTMLVideoElement> mainVideo;
+ WebCore::FloatRect mainVideoBounds;
+ for (auto& video : WebCore::descendantsOfType<WebCore::HTMLVideoElement>(*m_element)) {
+ auto rendererAndBounds = video.boundingAbsoluteRectWithoutLayout();
+ if (!rendererAndBounds)
+ continue;
+
+ auto [renderer, bounds] = *rendererAndBounds;
+ if (!renderer || bounds.isEmpty())
+ continue;
+
+ if (bounds.area() <= mainVideoBounds.area())
+ continue;
+
+ mainVideoBounds = bounds;
+ mainVideo = &video;
+ }
+ return mainVideo;
+ }());
+
+ if (m_mainVideoElement && m_mainVideoElement->paused())
+ scheduleMainVideoElementExtraction();
+#endif // ENABLE(VIDEO)
}
void WebFullScreenManager::willExitFullScreen()
@@ -268,6 +307,9 @@
m_closing = true;
LOG(Fullscreen, "WebFullScreenManager %p close()", this);
m_page->injectedBundleFullScreenClient().closeFullScreen(m_page.get());
+#if ENABLE(VIDEO)
+ setMainVideoElement(nullptr);
+#endif
m_closing = false;
}
@@ -296,6 +338,78 @@
m_page->corePage()->setFullscreenControlsHidden(hidden);
}
+void WebFullScreenManager::handleEvent(WebCore::ScriptExecutionContext& context, WebCore::Event& event)
+{
+#if ENABLE(VIDEO)
+ if (!m_mainVideoElement || event.target() != m_mainVideoElement || &context != &m_mainVideoElement->document())
+ return;
+
+ if (m_mainVideoElement->paused())
+ scheduleMainVideoElementExtraction();
+ else
+ endMainVideoElementExtractionIfNeeded();
+#else
+ UNUSED_PARAM(event);
+ UNUSED_PARAM(context);
+#endif
+}
+
+#if ENABLE(VIDEO)
+
+void WebFullScreenManager::mainVideoElementExtractionTimerFired()
+{
+ if (!m_mainVideoElement)
+ return;
+
+ m_isExtractingMainVideoElement = true;
+ // FIXME: Begin main video extraction in for element fullscreen.
+}
+
+void WebFullScreenManager::scheduleMainVideoElementExtraction()
+{
+ m_mainVideoElementExtractionTimer.startOneShot(250_ms);
+}
+
+void WebFullScreenManager::endMainVideoElementExtractionIfNeeded()
+{
+ m_mainVideoElementExtractionTimer.stop();
+
+ if (m_isExtractingMainVideoElement) {
+ // FIXME: Cancel the current video extraction session.
+ m_isExtractingMainVideoElement = false;
+ }
+}
+
+void WebFullScreenManager::setMainVideoElement(RefPtr<WebCore::HTMLVideoElement>&& element)
+{
+ if (element == m_mainVideoElement.get())
+ return;
+
+ // FIXME: We should listen for additional events (such as 'load' and 'unload') that bubble up
+ // to the fullscreen element, and recompute the main video element.
+ static NeverDestroyed eventsToObserve = std::array {
+ WebCore::eventNames().seekingEvent,
+ WebCore::eventNames().playingEvent,
+ WebCore::eventNames().pauseEvent,
+ };
+
+ if (m_mainVideoElement) {
+ for (auto& eventName : eventsToObserve.get())
+ m_mainVideoElement->removeEventListener(eventName, *this, { });
+
+ endMainVideoElementExtractionIfNeeded();
+ }
+
+ m_mainVideoElement = WTFMove(element);
+
+ if (m_mainVideoElement) {
+ for (auto& eventName : eventsToObserve.get())
+ m_mainVideoElement->addEventListener(eventName, *this, { });
+ }
+}
+
+#endif // ENABLE(VIDEO)
+
} // namespace WebKit
#endif // ENABLE(FULLSCREEN_API)
Modified: trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.h (292204 => 292205)
--- trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.h 2022-04-01 05:23:09 UTC (rev 292204)
+++ trunk/Source/WebKit/WebProcess/FullScreen/WebFullScreenManager.h 2022-04-01 05:52:48 UTC (rev 292205)
@@ -28,10 +28,13 @@
#if ENABLE(FULLSCREEN_API)
#include "WebCoreArgumentCoders.h"
+#include <WebCore/EventListener.h>
#include <WebCore/IntRect.h>
#include <WebCore/LengthBox.h>
#include <wtf/RefCounted.h>
#include <wtf/RefPtr.h>
+#include <wtf/RunLoop.h>
+#include <wtf/WeakPtr.h>
namespace IPC {
class Connection;
@@ -49,7 +52,7 @@
class WebPage;
-class WebFullScreenManager : public RefCounted<WebFullScreenManager> {
+class WebFullScreenManager final : public WebCore::EventListener {
public:
static Ref<WebFullScreenManager> create(WebPage*);
virtual ~WebFullScreenManager();
@@ -69,6 +72,8 @@
void videoControlsManagerDidChange();
+ bool operator==(const WebCore::EventListener& listener) const final { return this == &listener; }
+
protected:
WebFullScreenManager(WebPage*);
@@ -97,6 +102,20 @@
private:
void close();
+
+ void handleEvent(WebCore::ScriptExecutionContext&, WebCore::Event&) final;
+
+#if ENABLE(VIDEO)
+ void scheduleMainVideoElementExtraction();
+ void endMainVideoElementExtractionIfNeeded();
+ void mainVideoElementExtractionTimerFired();
+ void setMainVideoElement(RefPtr<WebCore::HTMLVideoElement>&&);
+
+ WeakPtr<WebCore::HTMLVideoElement> m_mainVideoElement;
+ RunLoop::Timer<WebFullScreenManager> m_mainVideoElementExtractionTimer;
+ bool m_isExtractingMainVideoElement { false };
+#endif // ENABLE(VIDEO)
+
bool m_closing { false };
};
