Title: [260245] trunk
Revision
260245
Author
[email protected]
Date
2020-04-17 07:22:23 -0700 (Fri, 17 Apr 2020)

Log Message

Safari doesn't apply frameRate limit when request stream from Camera
https://bugs.webkit.org/show_bug.cgi?id=210186
<rdar://problem/61452794>

Reviewed by Eric Carlson.

Source/WebCore:

Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.

Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
Fix mock capture source to update its frame rate when asked by applyConstraints.

Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-increasing.html

* platform/mediastream/RealtimeVideoCaptureSource.cpp:
(WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
(WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):
* platform/mediastream/RealtimeVideoCaptureSource.h:
(WebCore::RealtimeVideoCaptureSource::observedFrameRate const):
* platform/mediastream/RealtimeVideoSource.cpp:
(WebCore::RealtimeVideoSource::RealtimeVideoSource):
(WebCore::m_source):
(WebCore::RealtimeVideoSource::adaptVideoSample):
(WebCore::RealtimeVideoSource::videoSampleAvailable):
* platform/mediastream/RealtimeVideoSource.h:
* platform/mock/MockRealtimeVideoSource.cpp:
(WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):
* testing/Internals.cpp:
(WebCore::Internals::observeMediaStreamTrack):

LayoutTests:

* fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
* fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
* fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
* fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
* fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
* webrtc/routines.js:

Modified Paths

Added Paths

Diff

Modified: trunk/LayoutTests/ChangeLog (260244 => 260245)


--- trunk/LayoutTests/ChangeLog	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/LayoutTests/ChangeLog	2020-04-17 14:22:23 UTC (rev 260245)
@@ -1,3 +1,21 @@
+2020-04-17  Youenn Fablet  <[email protected]>
+
+        Safari doesn't apply frameRate limit when request stream from Camera
+        https://bugs.webkit.org/show_bug.cgi?id=210186
+        <rdar://problem/61452794>
+
+        Reviewed by Eric Carlson.
+
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
+        * fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
+        * webrtc/routines.js:
+
 2020-04-17  Alexey Shvayka  <[email protected]>
 
         MediaQueryList should extend EventTarget

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,4 @@
+
+
+PASS clone and applyConstraints to decrease frame rate 
+

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src=""
+    <script src=""
+    <script src =""
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+    const stream2 = stream1.clone();
+
+    await stream2.getVideoTracks()[0].applyConstraints({ width : 160, frameRate : 5 });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+
+    let frameRate = await computeFrameRate(stream1, video);
+    assert_greater_than(frameRate, 10, "stream1 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream1 frame rate below 60");
+
+    frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 1, "stream2 frame rate above 1");
+    assert_less_than(frameRate, 20, "stream2 frame rate below 20");
+}, "clone and applyConstraints to decrease frame rate");
+    </script>
+</body>
+</html>

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,4 @@
+
+
+PASS clone and applyConstraints to increase frame rate 
+

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src=""
+    <script src=""
+    <script src =""
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+    const stream2 = stream1.clone();
+
+    await stream2.getVideoTracks()[0].applyConstraints({ width : 720, frameRate : 30 });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+
+    let frameRate = await computeFrameRate(stream1, video);
+    assert_greater_than(frameRate, 1, "stream1 frame rate above 1");
+    assert_less_than(frameRate, 20, "stream1 frame rate below 20");
+
+    frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 10, "stream2 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream2 frame rate below 60");
+}, "clone and applyConstraints to increase frame rate");
+    </script>
+</body>
+</html>

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,4 @@
+
+
+PASS getUserMedia with decreasing frame rates 
+

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src=""
+    <script src=""
+    <script src =""
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+    const stream2 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+
+    if (!stream1.getVideoTracks()[0].muted) {
+        const frameRate = await computeFrameRate(stream1, video);
+        assert_greater_than(frameRate, 10, "stream1 frame rate above 10");
+        assert_less_than(frameRate, 60, "stream1 frame rate below 60");
+    }
+
+    const frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 1, "stream2 frame rate above 1");
+    assert_less_than(frameRate, 20, "stream2 frame rate below 20");
+}, "getUserMedia with decreasing frame rates");
+    </script>
+</body>
+</html>

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,4 @@
+
+
+PASS getUserMedia with increasing frame rates 
+

Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html (0 => 260245)


--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html	                        (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html	2020-04-17 14:22:23 UTC (rev 260245)
@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src=""
+    <script src=""
+    <script src =""
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+    const stream2 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+
+    if (!stream1.getVideoTracks()[0].muted) {
+        const frameRate = await computeFrameRate(stream1, video);
+        assert_greater_than(frameRate, 1, "stream1 frame rate above 1");
+        assert_less_than(frameRate, 20, "stream1 frame rate below 20");
+    }
+
+    const frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 10, "stream2 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream2 frame rate below 60");
+}, "getUserMedia with increasing frame rates");
+    </script>
+</body>
+</html>

Modified: trunk/LayoutTests/webrtc/routines.js (260244 => 260245)


--- trunk/LayoutTests/webrtc/routines.js	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/LayoutTests/webrtc/routines.js	2020-04-17 14:22:23 UTC (rev 260245)
@@ -225,3 +225,45 @@
     });
     return stats;
 }
+
+function getReceivedTrackStats(connection)
+{
+    return connection.getStats().then((report) => {
+        var stats;
+        report.forEach((statItem) => {
+            if (statItem.type === "track") {
+                stats = statItem;
+            }
+        });
+        return stats;
+    });
+}
+
+async function computeFrameRate(stream, video)
+{
+    if (window.internals) {
+        internals.observeMediaStreamTrack(stream.getVideoTracks()[0]);
+        await new Promise(resolve => setTimeout(resolve, 1000)); 
+        return internals.trackVideoSampleCount;
+    }
+
+    let connection;
+    video.srcObject = await new Promise((resolve, reject) => {
+        createConnections((firstConnection) => {
+            firstConnection.addTrack(stream.getVideoTracks()[0], stream);
+        }, (secondConnection) => {
+            connection = secondConnection;
+            secondConnection._ontrack_ = (trackEvent) => {
+                resolve(trackEvent.streams[0]);
+            };
+        });
+        setTimeout(() => reject("Test timed out"), 5000);
+    });
+
+    await video.play();
+
+    const stats1 = await getReceivedTrackStats(connection);
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+    const stats2 = await getReceivedTrackStats(connection);
+    return (stats2.framesReceived - stats1.framesReceived) * 1000 / (stats2.timestamp - stats1.timestamp);
+}

Modified: trunk/Source/WebCore/ChangeLog (260244 => 260245)


--- trunk/Source/WebCore/ChangeLog	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/ChangeLog	2020-04-17 14:22:23 UTC (rev 260245)
@@ -1,3 +1,39 @@
+2020-04-17  Youenn Fablet  <[email protected]>
+
+        Safari doesn't apply frameRate limit when request stream from Camera
+        https://bugs.webkit.org/show_bug.cgi?id=210186
+        <rdar://problem/61452794>
+
+        Reviewed by Eric Carlson.
+
+        Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
+        This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.
+
+        Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
+        Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
+        Fix mock capture source to update its frame rate when asked by applyConstraints.
+
+        Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-increasing.html
+
+        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
+        (WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
+        (WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):
+        * platform/mediastream/RealtimeVideoCaptureSource.h:
+        (WebCore::RealtimeVideoCaptureSource::observedFrameRate const):
+        * platform/mediastream/RealtimeVideoSource.cpp:
+        (WebCore::RealtimeVideoSource::RealtimeVideoSource):
+        (WebCore::m_source):
+        (WebCore::RealtimeVideoSource::adaptVideoSample):
+        (WebCore::RealtimeVideoSource::videoSampleAvailable):
+        * platform/mediastream/RealtimeVideoSource.h:
+        * platform/mock/MockRealtimeVideoSource.cpp:
+        (WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):
+        * testing/Internals.cpp:
+        (WebCore::Internals::observeMediaStreamTrack):
+
 2020-04-17  Alexey Shvayka  <[email protected]>
 
         MediaQueryList should extend EventTarget

Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp (260244 => 260245)


--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp	2020-04-17 14:22:23 UTC (rev 260245)
@@ -34,10 +34,6 @@
 #include "RemoteVideoSample.h"
 #include <wtf/JSONValues.h>
 
-#if PLATFORM(COCOA)
-#include "ImageTransferSessionVT.h"
-#endif
-
 namespace WebCore {
 
 RealtimeVideoCaptureSource::RealtimeVideoCaptureSource(String&& name, String&& id, String&& hashSalt)
@@ -376,7 +372,7 @@
     setFrameRate(match->requestedFrameRate);
 }
 
-RefPtr<MediaSample> RealtimeVideoCaptureSource::adaptVideoSample(MediaSample& sample)
+void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
 {
     MediaTime sampleTime = sample.presentationTime();
 
@@ -390,35 +386,9 @@
     if (interval > 1)
         m_observedFrameRate = (m_observedFrameTimeStamps.size() / interval);
 
-    auto mediaSample = makeRefPtr(&sample);
-
-#if PLATFORM(COCOA)
-    auto size = this->size();
-    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
-
-        if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
-            m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
-
-        ASSERT(m_imageTransferSession);
-        if (m_imageTransferSession) {
-            mediaSample = m_imageTransferSession->convertMediaSample(sample, size);
-            if (!mediaSample) {
-                ASSERT_NOT_REACHED();
-                return nullptr;
-            }
-        }
-    }
-#endif
-
-    return mediaSample.releaseNonNull();
+    videoSampleAvailable(sample);
 }
 
-void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
-{
-    if (auto mediaSample = adaptVideoSample(sample))
-        videoSampleAvailable(*mediaSample);
-}
-
 void RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate(Optional<int> width, Optional<int> height, Optional<double> frameRate)
 {
     // FIXME: We only change settings if capture resolution is below requested one. We should get the best preset for all clients.
@@ -427,9 +397,10 @@
         width = { };
     if (height && *height < static_cast<int>(settings.height()))
         height = { };
+    if (frameRate && *frameRate < static_cast<double>(settings.frameRate()))
+        frameRate = { };
 
-    // FIXME: handle frameRate potential increase.
-    if (!width && !height)
+    if (!width && !height && !frameRate)
         return;
 
     auto match = bestSupportedSizeAndFrameRate(width, height, frameRate);

Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h (260244 => 260245)


--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h	2020-04-17 14:22:23 UTC (rev 260245)
@@ -48,6 +48,8 @@
     virtual void generatePresets() = 0;
     virtual MediaSample::VideoRotation sampleRotation() const { return MediaSample::VideoRotation::None; }
 
+    double observedFrameRate() const { return m_observedFrameRate; }
+
 protected:
     RealtimeVideoCaptureSource(String&& name, String&& id, String&& hashSalt);
 
@@ -69,11 +71,8 @@
 
     void setDefaultSize(const IntSize& size) { m_defaultSize = size; }
 
-    double observedFrameRate() const { return m_observedFrameRate; }
-
     void dispatchMediaSampleToObservers(MediaSample&);
     const Vector<IntSize>& standardVideoSizes();
-    RefPtr<MediaSample> adaptVideoSample(MediaSample&);
 
 private:
     struct CaptureSizeAndFrameRate {
@@ -93,9 +92,6 @@
     Deque<double> m_observedFrameTimeStamps;
     double m_observedFrameRate { 0 };
     IntSize m_defaultSize;
-#if PLATFORM(COCOA)
-    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
-#endif
 };
 
 struct SizeAndFrameRate {

Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp (260244 => 260245)


--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp	2020-04-17 14:22:23 UTC (rev 260245)
@@ -28,15 +28,20 @@
 
 #if ENABLE(MEDIA_STREAM)
 
+#if PLATFORM(COCOA)
+#include "ImageTransferSessionVT.h"
+#endif
+
 namespace WebCore {
 
 RealtimeVideoSource::RealtimeVideoSource(Ref<RealtimeVideoCaptureSource>&& source)
-    : RealtimeVideoCaptureSource(String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
+    : RealtimeMediaSource(Type::Video, String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
     , m_source(WTFMove(source))
 {
     m_source->addObserver(*this);
     m_currentSettings = m_source->settings();
     setSize(m_source->size());
+    setFrameRate(m_source->frameRate());
 }
 
 RealtimeVideoSource::~RealtimeVideoSource()
@@ -142,13 +147,46 @@
     });
 }
 
+#if PLATFORM(COCOA)
+RefPtr<MediaSample> RealtimeVideoSource::adaptVideoSample(MediaSample& sample)
+{
+    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
+        m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
+
+    ASSERT(m_imageTransferSession);
+    if (!m_imageTransferSession)
+        return nullptr;
+
+    auto mediaSample = m_imageTransferSession->convertMediaSample(sample, size());
+    ASSERT(mediaSample);
+
+    return mediaSample;
+}
+#endif
+
 void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample)
 {
     if (!isProducingData())
         return;
 
-    if (auto mediaSample = adaptVideoSample(sample))
-        RealtimeMediaSource::videoSampleAvailable(*mediaSample);
+    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
+        return;
+
+    m_frameDecimation = static_cast<size_t>(m_source->observedFrameRate() / frameRate());
+    if (!m_frameDecimation)
+        m_frameDecimation = 1;
+
+#if PLATFORM(COCOA)
+    auto size = this->size();
+    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
+        if (auto mediaSample = adaptVideoSample(sample)) {
+            RealtimeMediaSource::videoSampleAvailable(*mediaSample);
+            return;
+        }
+    }
+#endif
+
+    RealtimeMediaSource::videoSampleAvailable(sample);
 }
 
 Ref<RealtimeMediaSource> RealtimeVideoSource::clone()

Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h (260244 => 260245)


--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h	2020-04-17 14:22:23 UTC (rev 260245)
@@ -31,8 +31,9 @@
 
 namespace WebCore {
 
-// FIXME: Make RealtimeVideoSource derive from RealtimeMediaSource directly.
-class RealtimeVideoSource final : public RealtimeVideoCaptureSource, public RealtimeMediaSource::Observer {
+class ImageTransferSessionVT;
+
+class RealtimeVideoSource final : public RealtimeMediaSource, public RealtimeMediaSource::Observer {
 public:
     static Ref<RealtimeVideoSource> create(Ref<RealtimeVideoCaptureSource>&& source) { return adoptRef(*new RealtimeVideoSource(WTFMove(source))); }
 
@@ -51,7 +52,6 @@
 
     const RealtimeMediaSourceCapabilities& capabilities() final { return m_source->capabilities(); }
     const RealtimeMediaSourceSettings& settings() final { return m_currentSettings; }
-    void generatePresets() final { m_source->generatePresets(); }
     bool isCaptureSource() const final { return m_source->isCaptureSource(); }
     CaptureDevice::DeviceType deviceType() const final { return m_source->deviceType(); }
     void monitorOrientation(OrientationNotifier& notifier) final { m_source->monitorOrientation(notifier); }
@@ -64,6 +64,10 @@
     bool preventSourceFromStopping() final;
     void videoSampleAvailable(MediaSample&) final;
 
+#if PLATFORM(COCOA)
+    RefPtr<MediaSample> adaptVideoSample(MediaSample&);
+#endif
+
 #if !RELEASE_LOG_DISABLED
     void setLogger(const Logger&, const void*) final;
 #endif
@@ -70,6 +74,11 @@
 
     Ref<RealtimeVideoCaptureSource> m_source;
     RealtimeMediaSourceSettings m_currentSettings;
+#if PLATFORM(COCOA)
+    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
+#endif
+    size_t m_frameDecimation { 1 };
+    size_t m_frameDecimationCounter { 0 };
 #if !RELEASE_LOG_DISABLED
     uint64_t m_cloneCounter { 0 };
 #endif

Modified: trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp (260244 => 260245)


--- trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp	2020-04-17 14:22:23 UTC (rev 260245)
@@ -187,11 +187,13 @@
     return m_currentSettings.value();
 }
 
-void MockRealtimeVideoSource::setFrameRateWithPreset(double, RefPtr<VideoPreset> preset)
+void MockRealtimeVideoSource::setFrameRateWithPreset(double frameRate, RefPtr<VideoPreset> preset)
 {
     m_preset = WTFMove(preset);
     if (m_preset)
         setIntrinsicSize(m_preset->size);
+    if (isProducingData())
+        m_emitFrameTimer.startRepeating(1_s / frameRate);
 }
 
 IntSize MockRealtimeVideoSource::captureSize() const

Modified: trunk/Source/WebCore/testing/Internals.cpp (260244 => 260245)


--- trunk/Source/WebCore/testing/Internals.cpp	2020-04-17 12:53:09 UTC (rev 260244)
+++ trunk/Source/WebCore/testing/Internals.cpp	2020-04-17 14:22:23 UTC (rev 260245)
@@ -5002,6 +5002,14 @@
 
 void Internals::observeMediaStreamTrack(MediaStreamTrack& track)
 {
+    if (m_trackSource) {
+        m_trackSource->removeObserver(*this);
+        m_trackSource->removeAudioSampleObserver(*this);
+
+        m_trackAudioSampleCount = 0;
+        m_trackVideoSampleCount = 0;
+    }
+
     m_trackSource = &track.source();
     m_trackSource->addObserver(*this);
     m_trackSource->addAudioSampleObserver(*this);
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to