Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 7db823fea8a1af033195fe8ddcaad6d7fae675b5
https://github.com/WebKit/WebKit/commit/7db823fea8a1af033195fe8ddcaad6d7fae675b5
Author: Jean-Yves Avenard <[email protected]>
Date: 2024-02-16 (Fri, 16 Feb 2024)
Changed paths:
M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
M Source/WTF/wtf/PlatformEnable.h
M Source/WTF/wtf/PlatformEnableCocoa.h
M Source/WebCore/CMakeLists.txt
M Source/WebCore/DerivedSources-input.xcfilelist
M Source/WebCore/DerivedSources-output.xcfilelist
M Source/WebCore/DerivedSources.make
M Source/WebCore/Headers.cmake
M Source/WebCore/Modules/mediasource/DOMURLMediaSource.cpp
M Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp
M Source/WebCore/Modules/mediasource/ManagedMediaSource.idl
M Source/WebCore/Modules/mediasource/MediaSource.cpp
M Source/WebCore/Modules/mediasource/MediaSource.h
M Source/WebCore/Modules/mediasource/MediaSource.idl
A Source/WebCore/Modules/mediasource/MediaSourceHandle.cpp
A Source/WebCore/Modules/mediasource/MediaSourceHandle.h
A Source/WebCore/Modules/mediasource/MediaSourceHandle.idl
A Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.cpp
A Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.h
A Source/WebCore/Modules/mediasource/MediaSourceInterfaceProxy.h
A Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.cpp
A Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.h
M Source/WebCore/Modules/mediasource/SourceBuffer.cpp
M Source/WebCore/Modules/mediasource/SourceBuffer.h
M Source/WebCore/Modules/mediasource/SourceBuffer.idl
M Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp
M Source/WebCore/Sources.txt
M Source/WebCore/WebCore.xcodeproj/project.pbxproj
M Source/WebCore/bindings/js/SerializedScriptValue.cpp
M Source/WebCore/bindings/js/SerializedScriptValue.h
M Source/WebCore/bindings/js/WebCoreBuiltinNames.h
M Source/WebCore/html/HTMLMediaElement.cpp
M Source/WebCore/html/HTMLMediaElement.h
M Source/WebCore/html/HTMLMediaElement.idl
M Source/WebCore/html/track/AudioTrack.cpp
M Source/WebCore/html/track/TrackListBase.cpp
M Source/WebCore/html/track/TrackListBase.h
M Source/WebCore/html/track/VideoTrack.cpp
M Source/WebCore/platform/graphics/MediaPlayer.cpp
M Source/WebCore/platform/graphics/MediaSourcePrivate.cpp
M Source/WebCore/platform/graphics/MediaSourcePrivate.h
M Source/WebCore/platform/graphics/MediaSourcePrivateClient.h
M Source/WebCore/platform/graphics/SourceBufferPrivate.cpp
M Source/WebCore/platform/graphics/SourceBufferPrivateClient.h
M Source/WebCore/platform/graphics/TrackPrivateBase.h
M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp
M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h
Log Message:
-----------
[MSE] Add MediaSourceHandle object and update to MSE's latest spec (including
MSE in worker)
https://bugs.webkit.org/show_bug.cgi?id=269461
rdar://problem/123006531
Reviewed by Youenn Fablet.
The MSE spec now defines that a MediaSource can be constructed in a
DedicatedWorker
and attached to a HTMLMediaElement running in the main thread through a
MediaSourceHandle
object.
To determine if a MediaSource can be constructed in a DedicatedWorker a new
attribute
has been added.
This commit adds support for all those latest functionalities and is placed
behind a MediaSourceInWorker
pref currently off by default.
Some architectural explanations:
The MediaSource and SourceBuffer were ThreadSafeRefCounted objects which was
problematic as we couldn't guarantee which thread those objets would be
destructed on.
To solve this, we make MediaSource and SourceBuffer be RefCounted instead and
use a ThreadSafeRefCounted {MediaSource|SourceBuffer}PrivateClient each
providing
a function dispatcher used for the {MediaSource|SourceBuffer}Private to
communicate
with their client.
The function dispatcher guarantees that:
1- The function will always be run on the MediaSource's thread.
2- That the MediaSource hasn't been destructed in-between calls
3- That the MediaSource's readyState is not "Closed".
As such, {MediaSource|SourceBuffer} methods are only ever called on the correct
thread,
it is no longer the responsibility of either the MediaSource or the
SourceBuffer to
run on the correct thread, so we can remove the now redundant
`assertIsCurrent(m_dispatcher)`
and the m_dispatcher member.
Guarantees 2 and 3 allows to remove the checks made in each method, allowing to
simplify the code.
One of the existing complexity, was that the media element directly interacted
with the MediaSource
through both sync and asynchronous methods which is no longer a possibility if
the MediaSource
now exists in the DedicatedWorker.
We introduce the MediaSourceInterfaceProxy virtual interface so that the media
element
can continue to interact with the MediaSource with the minimum amount of
changes.
Two implementation are provided.
MediaSourceInterfaceMainThread, which performs a simple, passthrough connection
and the MediaSourceInterfaceWorker which is constructed from the
MediaSourceHandle.
The MediaSourceHandle implementation contains the minimum information so that
the
MediaSourceInterfaceProxy can be established.
While asynchronous calls can be easily achieved with the MediaSourceHandle's
function dispatcher, synchronous calls can't (you can't dispatch synchronously
to a worker)
As such, all information needed to be synchronously retrieved
by the HTMLMediaElement from the MediaSource are now moved to the
MediaSourcePrivate
instead, this includes:
- readyState
- duration
- buffered range
- seekable range
- live seekable range.
In order to avoid duplicating such handling with all MediaSourcePrivate
implementations:
(AVF, Mock, GStreamer, GPU) we place it in the MediaSourcePrivate base class.
Various small changes had to be performed for things to operate when a worker
is in use.
- Tracks validity where being checked using main thread's only AtomString, we
change the
code use litteral strings for now.
- Some calls from the MediaSource (worker) are made directly to the MediaPlayer
(which
lives in the main thread), rather than making the code thread-safe, we instead
dispatch synchronously to the main thread.
Present issues:
- The handling of Tracks and TrackList in the worker is currently broken as the
MSE spec do not cater yet for necessary changes (such as {Audio|Video|Text}Track
objects are currently only exposed to the `Document` (see
https://github.com/w3c/media-source/issues/280)
- Media Logging in the Worker is currently disabled.
- There appear to still be some unnecessary hops via the main thread while a
MediaSource is in
operation in the work, which results in some janks still occurring when the
main thread is
under heavy load.
- When in the main thread, it is possible via dynamic preferences to limit the
codec's supported
configuration. As Settings aren't available in the Worker, this functionality
is disabled.
MediaSource in a Worker functionality is currently turn off by default.
Tests will be added at a later stage.
No changes in observable behaviour, currently covered by extensive existing
tests.
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WTF/wtf/PlatformEnable.h:
* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WebCore/CMakeLists.txt:
* Source/WebCore/DerivedSources-input.xcfilelist:
* Source/WebCore/DerivedSources-output.xcfilelist:
* Source/WebCore/DerivedSources.make:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/mediasource/DOMURLMediaSource.cpp:
(WebCore::DOMURLMediaSource::createObjectURL):
* Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp:
(WebCore::ManagedMediaSource::setStreaming):
(WebCore::ManagedMediaSource::ensurePrefsRead):
(WebCore::ManagedMediaSource::monitorSourceBuffers):
(WebCore::ManagedMediaSource::streamingTimerFired):
* Source/WebCore/Modules/mediasource/ManagedMediaSource.idl:
* Source/WebCore/Modules/mediasource/MediaSource.cpp:
(WebCore::MediaSource::setRegistry):
(WebCore::MediaSource::MediaSource):
(WebCore::MediaSource::logger):
(WebCore::MediaSource::didLogMessage):
(WebCore::MediaSource::setPrivateAndOpen):
(WebCore::MediaSource::currentTime const):
(WebCore::MediaSource::buffered const):
(WebCore::MediaSource::waitForTarget):
(WebCore::MediaSource::completeSeek):
(WebCore::MediaSource::seekToTime):
(WebCore::MediaSource::seekable):
(WebCore::MediaSource::setLiveSeekableRange):
(WebCore::MediaSource::clearLiveSeekableRange):
(WebCore::MediaSource::hasBufferedTime):
(WebCore::MediaSource::hasFutureTime):
(WebCore::MediaSource::monitorSourceBuffers):
(WebCore::MediaSource::setDuration):
(WebCore::MediaSource::setDurationInternal):
(WebCore::MediaSource::setReadyState):
(WebCore::MediaSource::endOfStream):
(WebCore::MediaSource::streamEndedWithError):
(WebCore::MediaSource::addSourceBuffer):
(WebCore::MediaSource::removeSourceBuffer):
(WebCore::MediaSource::isTypeSupported):
(WebCore::MediaSource::isOpen const):
(WebCore::MediaSource::isClosed const):
(WebCore::MediaSource::isEnded const):
(WebCore::MediaSource::detachFromElement):
(WebCore::MediaSource::attachToElement):
(WebCore::MediaSource::openIfInEndedState):
(WebCore::MediaSource::openIfDeferredOpen):
(WebCore::MediaSource::stop):
(WebCore::MediaSource::readyState const):
(WebCore::MediaSource::onReadyStateChange):
(WebCore::MediaSource::activeRanges const):
(WebCore::MediaSource::regenerateActiveSourceBuffers):
(WebCore::MediaSource::notifyElementUpdateMediaState const):
(WebCore::MediaSource::ensureWeakOnHTMLMediaElementContext const):
(WebCore::MediaSource::sourceBufferBufferedChanged):
(WebCore::MediaSource::updateBufferedIfNeeded):
(WebCore::MediaSource::failedToCreateRenderer):
(WebCore::MediaSource::sourceBufferReceivedFirstInitializationSegmentChanged):
(WebCore::MediaSource::sourceBufferActiveTrackFlagChanged):
(WebCore::MediaSource::setMediaPlayerReadyState):
(WebCore::MediaSource::incrementDroppedFrameCount):
(WebCore::MediaSource::addAudioTrackToElement):
(WebCore::MediaSource::addTextTrackToElement):
(WebCore::MediaSource::addVideoTrackToElement):
(WebCore::MediaSource::addAudioTrackMirrorToElement):
(WebCore::MediaSource::addTextTrackMirrorToElement):
(WebCore::MediaSource::addVideoTrackMirrorToElement):
(WebCore::MediaSource::memoryPressure):
(WebCore::MediaSource::client const):
(WebCore::MediaSource::enabledForContext):
(WebCore::MediaSource::handle):
(WebCore::MediaSource::canConstructInDedicatedWorker):
(WebCore::MediaSource::settings const): Deleted.
(WebCore::MediaSource::ensureWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/MediaSource.h:
(WebCore::MediaSource::promisedWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/MediaSource.idl:
* Source/WebCore/Modules/mediasource/MediaSourceHandle.cpp: Added.
(WebCore::MediaSourceHandle::create):
(WebCore::MediaSourceHandle::MediaSourceHandle):
(WebCore::MediaSourceHandle::canDetach const):
(WebCore::MediaSourceHandle::setHasEverBeenAssignedAsSrcObject):
(WebCore::MediaSourceHandle::hasEverBeenAssignedAsSrcObject const):
(WebCore::MediaSourceHandle::isManaged const):
(WebCore::MediaSourceHandle::ensureOnDispatcher const):
(WebCore::MediaSourceHandle::detach):
(WebCore::MediaSourceHandle::mediaSourcePrivateClient const):
(WebCore::MediaSourceHandle::mediaSourceDidOpen):
(WebCore::MediaSourceHandle::mediaSourcePrivate const):
* Source/WebCore/Modules/mediasource/MediaSourceHandle.h: Added.
(WebCore::MediaSourceHandle::isDetached const):
(WebCore::MediaSourceHandle::setDetached):
* Source/WebCore/Modules/mediasource/MediaSourceHandle.idl: Copied from
Source/WebCore/Modules/mediasource/ManagedMediaSource.idl.
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.cpp: Added.
(WebCore::MediaSourceInterfaceMainThread::MediaSourceInterfaceMainThread):
(WebCore::MediaSourceInterfaceMainThread::client const):
(WebCore::MediaSourceInterfaceMainThread::monitorSourceBuffers):
(WebCore::MediaSourceInterfaceMainThread::isClosed const):
(WebCore::MediaSourceInterfaceMainThread::duration const):
(WebCore::MediaSourceInterfaceMainThread::buffered const):
(WebCore::MediaSourceInterfaceMainThread::seekable const):
(WebCore::MediaSourceInterfaceMainThread::isStreamingContent const):
(WebCore::MediaSourceInterfaceMainThread::attachToElement):
(WebCore::MediaSourceInterfaceMainThread::detachFromElement):
(WebCore::MediaSourceInterfaceMainThread::openIfDeferredOpen):
(WebCore::MediaSourceInterfaceMainThread::isManaged const):
(WebCore::MediaSourceInterfaceMainThread::setAsSrcObject):
(WebCore::MediaSourceInterfaceMainThread::memoryPressure):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.h: Copied
from Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
(WebCore::MediaSourceInterfaceMainThread::create):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceProxy.h: Copied from
Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.cpp: Added.
(WebCore::MediaSourceInterfaceWorker::MediaSourceInterfaceWorker):
(WebCore::MediaSourceInterfaceWorker::client const):
(WebCore::MediaSourceInterfaceWorker::monitorSourceBuffers):
(WebCore::MediaSourceInterfaceWorker::isClosed const):
(WebCore::MediaSourceInterfaceWorker::duration const):
(WebCore::MediaSourceInterfaceWorker::buffered const):
(WebCore::MediaSourceInterfaceWorker::seekable const):
(WebCore::MediaSourceInterfaceWorker::isStreamingContent const):
(WebCore::MediaSourceInterfaceWorker::attachToElement):
(WebCore::MediaSourceInterfaceWorker::detachFromElement):
(WebCore::MediaSourceInterfaceWorker::openIfDeferredOpen):
(WebCore::MediaSourceInterfaceWorker::isManaged const):
(WebCore::MediaSourceInterfaceWorker::setAsSrcObject):
(WebCore::MediaSourceInterfaceWorker::memoryPressure):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.h: Copied from
Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
(WebCore::MediaSourceInterfaceWorker::create):
* Source/WebCore/Modules/mediasource/SourceBuffer.cpp:
(WebCore::SourceBuffer::SourceBuffer):
(WebCore::m_logIdentifier):
(WebCore::SourceBuffer::~SourceBuffer):
(WebCore::SourceBuffer::abort):
(WebCore::SourceBuffer::rangeRemoval):
(WebCore::SourceBuffer::changeType):
(WebCore::SourceBuffer::abortIfUpdating):
(WebCore::SourceBuffer::removedFromMediaSource):
(WebCore::SourceBuffer::seekToTime):
(WebCore::SourceBuffer::appendBufferInternal):
(WebCore::SourceBuffer::sourceBufferPrivateAppendComplete):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveRenderingError):
(WebCore::SourceBuffer::maximumBufferSize const):
(WebCore::SourceBuffer::setActive):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveInitializationSegment):
(WebCore::SourceBuffer::validateInitializationSegment):
(WebCore::SourceBuffer::sourceBufferPrivateDurationChanged):
(WebCore::SourceBuffer::sourceBufferPrivateHighestPresentationTimestampChanged):
(WebCore::SourceBuffer::sourceBufferPrivateDidDropSample):
(WebCore::SourceBuffer::bufferedSamplesForTrackId):
(WebCore::SourceBuffer::enqueuedSamplesForTrackID):
(WebCore::SourceBuffer::sourceBufferPrivateBufferedChanged):
(WebCore::SourceBuffer::enabledForContext):
(WebCore::SourceBuffer::settings const): Deleted.
(WebCore::SourceBuffer::ensureWeakOnDispatcher const): Deleted.
(WebCore::SourceBuffer::promisedWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/SourceBuffer.h:
* Source/WebCore/Modules/mediasource/SourceBuffer.idl:
* Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/bindings/js/SerializedScriptValue.cpp:
(WebCore::name):
(WebCore::isTypeExposedToGlobalObject):
(WebCore::CloneSerializer::serialize):
(WebCore::CloneSerializer::CloneSerializer):
(WebCore::CloneSerializer::dumpMediaSourceHandle):
(WebCore::CloneSerializer::dumpIfTerminal):
(WebCore::CloneDeserializer::deserialize):
(WebCore::CloneDeserializer::CloneDeserializer):
(WebCore::CloneDeserializer::takeDetachedMediaSourceHandles):
(WebCore::CloneDeserializer::readMediaSourceHandle):
(WebCore::CloneDeserializer::readTerminal):
(WebCore::validateSerializedResult):
(WebCore::SerializedScriptValue::SerializedScriptValue):
(WebCore::canDetachMediaSourceHandles):
(WebCore::SerializedScriptValue::create):
(WebCore::SerializedScriptValue::deserialize):
* Source/WebCore/bindings/js/SerializedScriptValue.h:
* Source/WebCore/bindings/js/WebCoreBuiltinNames.h:
* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::selectMediaResource):
(WebCore::HTMLMediaElement::loadResource):
(WebCore::HTMLMediaElement::hasManagedMediaSource const):
(WebCore::HTMLMediaElement::detachMediaSource):
(WebCore::HTMLMediaElement::mediaState const):
* Source/WebCore/html/HTMLMediaElement.h:
* Source/WebCore/html/HTMLMediaElement.idl:
* Source/WebCore/html/track/AudioTrack.cpp:
(WebCore::AudioTrack::isValidKind const):
(WebCore::AudioTrack::updateKindFromPrivate):
* Source/WebCore/html/track/VideoTrack.cpp:
(WebCore::VideoTrack::isValidKind const):
(WebCore::VideoTrack::updateKindFromPrivate):
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::applicationOctetStream):
* Source/WebCore/platform/graphics/MediaSourcePrivate.cpp:
(WebCore::MediaSourcePrivate::MediaSourcePrivate):
(WebCore::MediaSourcePrivate::seekable const):
(WebCore::MediaSourcePrivate::setLiveSeekableRange):
(WebCore::MediaSourcePrivate::clearLiveSeekableRange):
(WebCore::MediaSourcePrivate::liveSeekableRange const):
* Source/WebCore/platform/graphics/MediaSourcePrivate.h:
* Source/WebCore/platform/graphics/MediaSourcePrivateClient.h:
(WebCore::MediaSourcePrivateClient::logger const):
* Source/WebCore/platform/graphics/SourceBufferPrivate.cpp:
(WebCore::SourceBufferPrivate::setClient):
* Source/WebCore/platform/graphics/SourceBufferPrivateClient.h:
* Source/WebCore/platform/graphics/TrackPrivateBase.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp:
(WebKit::MediaSourcePrivateRemote::MediaSourcePrivateRemote):
(WebKit::MediaSourcePrivateRemote::mediaPlayerReadyState const):
(WebKit::MediaSourcePrivateRemote::setMediaPlayerReadyState):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::mediaSourcePrivateShuttingDown):
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h:
Canonical link: https://commits.webkit.org/274853@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes
