Log Message
[GStreamer] Support arbitrary video resolution in getUserMedia API https://bugs.webkit.org/show_bug.cgi?id=189734
Source/WebCore: Patch by Thibault Saunier <[email protected]> on 2018-10-11 Reviewed by Xabier Rodriguez-Calvar. Implement arbitrary video resolution for the getUserMedia API in GStreamer. Fix the MockRealtimeMediaSource device list to make devices properties match test expectations. Reactivate tests that were failling because of that. * platform/mediastream/RealtimeVideoSource.cpp: (WebCore::RealtimeVideoSource::standardVideoSizes): (WebCore::standardVideoSizes): Deleted. * platform/mediastream/RealtimeVideoSource.h: * platform/mediastream/VideoPreset.h: * platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp: (WebCore::GStreamerVideoPreset::create): (WebCore::GStreamerVideoPreset::GStreamerVideoPreset): (WebCore::GStreamerVideoCaptureSource::GStreamerVideoCaptureSource): (WebCore::GStreamerVideoCaptureSource::capabilities): (WebCore::GStreamerVideoCaptureSource::generatePresets): * platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h: * platform/mock/MockRealtimeMediaSourceCenter.cpp: (WebCore::defaultDevices): LayoutTests: Implement arbitrary video resolution for the getUserMedia API in GStreamer. Fix the MockRealtimeMediaSource device list to make devices properties match test expectations. Reactivate tests that were failling because of that. Patch by Thibault Saunier <[email protected]> on 2018-10-11 Reviewed by Xabier Rodriguez-Calvar. * platform/gtk/TestExpectations: * platform/wpe/TestExpectations:
Modified Paths
- trunk/LayoutTests/ChangeLog
- trunk/LayoutTests/platform/gtk/TestExpectations
- trunk/LayoutTests/platform/wpe/TestExpectations
- trunk/Source/WebCore/ChangeLog
- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp
- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h
- trunk/Source/WebCore/platform/mediastream/VideoPreset.h
- trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp
- trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h
- trunk/Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp
Diff
Modified: trunk/LayoutTests/ChangeLog (237047 => 237048)
--- trunk/LayoutTests/ChangeLog 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/LayoutTests/ChangeLog 2018-10-11 21:30:33 UTC (rev 237048)
@@ -1,3 +1,20 @@
+2018-10-11 Thibault Saunier <[email protected]>
+
+ [GStreamer] Support arbitrary video resolution in getUserMedia API
+ https://bugs.webkit.org/show_bug.cgi?id=189734
+
+ Implement arbitrary video resolution for the getUserMedia API in GStreamer.
+
+ Fix the MockRealtimeMediaSource device list to make devices properties match
+ test expectations.
+
+ Reactivate tests that were failling because of that.
+
+ Reviewed by Xabier Rodriguez-Calvar.
+
+ * platform/gtk/TestExpectations:
+ * platform/wpe/TestExpectations:
+
2018-10-11 Per Arne Vollan <[email protected]>
Layout Test fast/forms/fieldset/fieldset-elements-htmlcollection.html is failing
Modified: trunk/LayoutTests/platform/gtk/TestExpectations (237047 => 237048)
--- trunk/LayoutTests/platform/gtk/TestExpectations 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/LayoutTests/platform/gtk/TestExpectations 2018-10-11 21:30:33 UTC (rev 237048)
@@ -611,10 +611,6 @@
http/tests/webrtc [ Skip ]
# The MediaStream implementation is also still not completed
webkit.org/b/79203 fast/mediastream/mock-media-source-webaudio.html [ Timeout ]
-webkit.org/b/189734 fast/mediastream/MediaStreamTrack-getCapabilities.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/apply-constraints-advanced.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/apply-constraints-video.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/getUserMedia-default.html [ Failure ]
webkit.org/b/79203 fast/mediastream/getUserMedia-webaudio.html [ Failure ]
# Crash is bug #186678
webkit.org/b/79203 fast/mediastream/MediaStream-video-element-track-stop.html [ Timeout Crash ]
Modified: trunk/LayoutTests/platform/wpe/TestExpectations (237047 => 237048)
--- trunk/LayoutTests/platform/wpe/TestExpectations 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/LayoutTests/platform/wpe/TestExpectations 2018-10-11 21:30:33 UTC (rev 237048)
@@ -46,6 +46,7 @@
#////////////////////////////////////////////////////////////////////////////////////////
# The webrtc implementation is not fully completed yet
+webkit.org/b/189567 webrtc/video-addLegacyTransceiver.html [ Failure ]
webkit.org/b/187064 webrtc/audio-peer-connection-g722.html
webkit.org/b/187064 webrtc/video-with-receiver.html
webkit.org/b/187064 webrtc/captureCanvas-webrtc.html
@@ -82,10 +83,6 @@
webkit.org/b/79203 fast/mediastream/mock-media-source-webaudio.html [ Timeout ]
webkit.org/b/79203 fast/mediastream/getUserMedia-webaudio.html [ Failure Crash ]
webkit.org/b/79203 fast/mediastream/MediaStream-video-element-track-stop.html [ Timeout ]
-webkit.org/b/189734 fast/mediastream/MediaStreamTrack-getCapabilities.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/apply-constraints-advanced.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/apply-constraints-video.html [ Failure ]
-webkit.org/b/189734 fast/mediastream/getUserMedia-default.html [ Failure ]
webkit.org/b/79203 fast/mediastream/RTCPeerConnection-dtmf.html [ Timeout ]
webkit.org/b/79203 fast/mediastream/RTCPeerConnection-icecandidate-event.html [ Pass Failure Crash ]
webkit.org/b/79203 fast/mediastream/RTCPeerConnection-media-setup-two-dialogs.html [ Pass Failure Crash ]
Modified: trunk/Source/WebCore/ChangeLog (237047 => 237048)
--- trunk/Source/WebCore/ChangeLog 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/ChangeLog 2018-10-11 21:30:33 UTC (rev 237048)
@@ -1,3 +1,32 @@
+2018-10-11 Thibault Saunier <[email protected]>
+
+ [GStreamer] Support arbitrary video resolution in getUserMedia API
+ https://bugs.webkit.org/show_bug.cgi?id=189734
+
+ Reviewed by Xabier Rodriguez-Calvar.
+
+ Implement arbitrary video resolution for the getUserMedia API in GStreamer.
+
+ Fix the MockRealtimeMediaSource device list to make devices properties match
+ test expectations.
+
+ Reactivate tests that were failling because of that.
+
+ * platform/mediastream/RealtimeVideoSource.cpp:
+ (WebCore::RealtimeVideoSource::standardVideoSizes):
+ (WebCore::standardVideoSizes): Deleted.
+ * platform/mediastream/RealtimeVideoSource.h:
+ * platform/mediastream/VideoPreset.h:
+ * platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp:
+ (WebCore::GStreamerVideoPreset::create):
+ (WebCore::GStreamerVideoPreset::GStreamerVideoPreset):
+ (WebCore::GStreamerVideoCaptureSource::GStreamerVideoCaptureSource):
+ (WebCore::GStreamerVideoCaptureSource::capabilities):
+ (WebCore::GStreamerVideoCaptureSource::generatePresets):
+ * platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h:
+ * platform/mock/MockRealtimeMediaSourceCenter.cpp:
+ (WebCore::defaultDevices):
+
2018-10-11 Keith Rollin <[email protected]>
CURRENT_ARCH should not be used in Run Script phase.
Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp (237047 => 237048)
--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp 2018-10-11 21:30:33 UTC (rev 237048)
@@ -91,7 +91,7 @@
}
}
-static const Vector<IntSize>& standardVideoSizes()
+const Vector<IntSize>& RealtimeVideoSource::standardVideoSizes()
{
static const auto sizes = makeNeverDestroyed([] {
static IntSize videoSizes[] = {
Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h (237047 => 237048)
--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h 2018-10-11 21:30:33 UTC (rev 237048)
@@ -67,6 +67,7 @@
double observedFrameRate() const { return m_observedFrameRate; }
void dispatchMediaSampleToObservers(MediaSample&);
+ const Vector<IntSize>& standardVideoSizes();
private:
struct CaptureSizeAndFrameRate {
Modified: trunk/Source/WebCore/platform/mediastream/VideoPreset.h (237047 => 237048)
--- trunk/Source/WebCore/platform/mediastream/VideoPreset.h 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mediastream/VideoPreset.h 2018-10-11 21:30:33 UTC (rev 237048)
@@ -107,7 +107,8 @@
enum VideoPresetType {
Base,
- AVCapture
+ AVCapture,
+ GStreamer
};
IntSize size;
Modified: trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp (237047 => 237048)
--- trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp 2018-10-11 21:30:33 UTC (rev 237048)
@@ -52,6 +52,19 @@
});
}
+class GStreamerVideoPreset : public VideoPreset {
+public:
+ static Ref<GStreamerVideoPreset> create(IntSize size, Vector<FrameRateRange>&& framerates)
+ {
+ return adoptRef(*new GStreamerVideoPreset(size, WTFMove(framerates)));
+ }
+
+ GStreamerVideoPreset(IntSize size, Vector<FrameRateRange>&& frameRateRanges)
+ : VideoPreset(size, WTFMove(frameRateRanges), GStreamer)
+ {
+ }
+};
+
class GStreamerVideoCaptureSourceFactory final : public VideoCaptureFactory {
public:
CaptureSourceOrError createVideoCaptureSource(const CaptureDevice& device, String&& hashSalt, const MediaConstraints* constraints) final
@@ -110,7 +123,7 @@
}
GStreamerVideoCaptureSource::GStreamerVideoCaptureSource(String&& deviceID, String&& name, String&& hashSalt, const gchar *source_factory)
- : RealtimeMediaSource(RealtimeMediaSource::Type::Video, WTFMove(deviceID), WTFMove(name), WTFMove(hashSalt))
+ : RealtimeVideoSource(WTFMove(deviceID), WTFMove(name), WTFMove(hashSalt))
, m_capturer(std::make_unique<GStreamerVideoCapturer>(source_factory))
{
initializeGStreamerDebug();
@@ -117,7 +130,7 @@
}
GStreamerVideoCaptureSource::GStreamerVideoCaptureSource(GStreamerCaptureDevice device, String&& hashSalt)
- : RealtimeMediaSource(RealtimeMediaSource::Type::Video, String { device.persistentId() }, String { device.label() }, WTFMove(hashSalt))
+ : RealtimeVideoSource(String { device.persistentId() }, String { device.label() }, WTFMove(hashSalt))
, m_capturer(std::make_unique<GStreamerVideoCapturer>(device))
{
initializeGStreamerDebug();
@@ -168,87 +181,13 @@
const RealtimeMediaSourceCapabilities& GStreamerVideoCaptureSource::capabilities()
{
- if (m_capabilities)
- return m_capabilities.value();
-
RealtimeMediaSourceCapabilities capabilities(settings().supportedConstraints());
- GRefPtr<GstCaps> caps = adoptGRef(m_capturer->caps());
- int32_t minWidth = G_MAXINT32, maxWidth = 0, minHeight = G_MAXINT32, maxHeight = 0;
- double minFramerate = G_MAXDOUBLE, maxFramerate = G_MINDOUBLE;
- for (guint i = 0; i < gst_caps_get_size(caps.get()); i++) {
- GstStructure* str = gst_caps_get_structure(caps.get(), i);
+ capabilities.setDeviceId(hashedId());
+ updateCapabilities(capabilities);
- // Only accept raw video for now.
- if (!gst_structure_has_name(str, "video/x-raw"))
- continue;
+ m_capabilities = WTFMove(capabilities);
- int32_t tmpMinWidth, tmpMinHeight, tmpMinFPSNumerator, tmpMinFPSDenominator;
- int32_t tmpMaxWidth, tmpMaxHeight, tmpMaxFPSNumerator, tmpMaxFPSDenominator;
- double tmpMinFramerate = G_MAXDOUBLE, tmpMaxFramerate = G_MINDOUBLE;
-
- if (!gst_structure_get(str, "width", GST_TYPE_INT_RANGE, &tmpMinWidth, &tmpMaxWidth, "height", GST_TYPE_INT_RANGE, &tmpMinHeight, &tmpMaxHeight, nullptr)) {
- if (!gst_structure_get(str, "width", G_TYPE_INT, &tmpMinWidth, "height", G_TYPE_INT, &tmpMinHeight, nullptr))
- continue;
-
- tmpMaxWidth = tmpMinWidth;
- tmpMaxHeight = tmpMinHeight;
- }
-
- if (gst_structure_get(str, "framerate", GST_TYPE_FRACTION_RANGE, &tmpMinFPSNumerator, &tmpMinFPSDenominator, &tmpMaxFPSNumerator, &tmpMaxFPSDenominator, nullptr)) {
- gst_util_fraction_to_double(tmpMinFPSNumerator, tmpMinFPSDenominator, &tmpMinFramerate);
- gst_util_fraction_to_double(tmpMaxFPSNumerator, tmpMaxFPSDenominator, &tmpMaxFramerate);
- } else if (gst_structure_get(str,
- "framerate", GST_TYPE_FRACTION, &tmpMinFPSNumerator, &tmpMinFPSDenominator, nullptr)) {
- tmpMaxFPSNumerator = tmpMinFPSNumerator;
- tmpMaxFPSDenominator = tmpMinFPSDenominator;
- gst_util_fraction_to_double(tmpMinFPSNumerator, tmpMinFPSDenominator, &tmpMinFramerate);
- gst_util_fraction_to_double(tmpMaxFPSNumerator, tmpMaxFPSDenominator, &tmpMaxFramerate);
- } else {
- const GValue* frameRates(gst_structure_get_value(str, "framerate"));
- tmpMinFramerate = G_MAXDOUBLE;
- tmpMaxFramerate = 0.0;
-
- guint frameRatesLength = static_cast<guint>(gst_value_list_get_size(frameRates)) - 1;
-
- for (guint i = 0; i < frameRatesLength; i++) {
- double tmpFrameRate;
- const GValue* val = gst_value_list_get_value(frameRates, i);
-
- ASSERT(G_VALUE_TYPE(val) == GST_TYPE_FRACTION);
- gst_util_fraction_to_double(gst_value_get_fraction_numerator(val),
- gst_value_get_fraction_denominator(val), &tmpFrameRate);
-
- tmpMinFramerate = std::min(tmpMinFramerate, tmpFrameRate);
- tmpMaxFramerate = std::max(tmpMaxFramerate, tmpFrameRate);
- }
-
- if (i > 0) {
- minWidth = std::min(tmpMinWidth, minWidth);
- minHeight = std::min(tmpMinHeight, minHeight);
- minFramerate = std::min(tmpMinFramerate, minFramerate);
-
- maxWidth = std::max(tmpMaxWidth, maxWidth);
- maxHeight = std::max(tmpMaxHeight, maxHeight);
- maxFramerate = std::max(tmpMaxFramerate, maxFramerate);
- } else {
- minWidth = tmpMinWidth;
- minHeight = tmpMinHeight;
- minFramerate = tmpMinFramerate;
-
- maxWidth = tmpMaxWidth;
- maxHeight = tmpMaxHeight;
- maxFramerate = tmpMaxFramerate;
- }
- }
-
- capabilities.setDeviceId(hashedId());
- capabilities.setWidth(CapabilityValueOrRange(minWidth, maxWidth));
- capabilities.setHeight(CapabilityValueOrRange(minHeight, maxHeight));
- capabilities.setFrameRate(CapabilityValueOrRange(minFramerate, maxFramerate));
- m_capabilities = WTFMove(capabilities);
- }
-
return m_capabilities.value();
}
@@ -278,6 +217,69 @@
return m_currentSettings.value();
}
+void GStreamerVideoCaptureSource::generatePresets()
+{
+ Vector<Ref<VideoPreset>> presets;
+ GRefPtr<GstCaps> caps = adoptGRef(m_capturer->caps());
+ for (unsigned i = 0; i < gst_caps_get_size(caps.get()); i++) {
+ GstStructure* str = gst_caps_get_structure(caps.get(), i);
+
+ // Only accept raw video for now.
+ if (!gst_structure_has_name(str, "video/x-raw"))
+ continue;
+
+ int32_t width, height;
+ if (!gst_structure_get(str, "width", G_TYPE_INT, &width, "height", G_TYPE_INT, &height, nullptr)) {
+ GST_INFO("Could not find discret height and width values in %" GST_PTR_FORMAT, str);
+ continue;
+ }
+
+ IntSize size = { width, height };
+ double framerate;
+ Vector<FrameRateRange> frameRates;
+ int32_t minFrameRateNumerator, minFrameRateDenominator, maxFrameRateNumerator, maxFrameRateDenominator, framerateNumerator, framerateDenominator;
+ if (gst_structure_get(str, "framerate", GST_TYPE_FRACTION_RANGE, &minFrameRateNumerator, &minFrameRateDenominator, &maxFrameRateNumerator, &maxFrameRateDenominator, nullptr)) {
+ FrameRateRange range;
+
+ gst_util_fraction_to_double(minFrameRateNumerator, minFrameRateDenominator, &range.minimum);
+ gst_util_fraction_to_double(maxFrameRateNumerator, maxFrameRateDenominator, &range.maximum);
+
+ frameRates.append(range);
+ } else if (gst_structure_get(str, "framerate", GST_TYPE_FRACTION, &framerateNumerator, &framerateDenominator, nullptr)) {
+ gst_util_fraction_to_double(framerateNumerator, framerateDenominator, &framerate);
+ frameRates.append({ framerate, framerate});
+ } else {
+ const GValue* frameRateValues(gst_structure_get_value(str, "framerate"));
+ unsigned frameRatesLength = static_cast<unsigned>(gst_value_list_get_size(frameRateValues));
+
+ for (unsigned j = 0; j < frameRatesLength; j++) {
+ const GValue* val = gst_value_list_get_value(frameRateValues, j);
+
+ ASSERT(val && G_VALUE_TYPE(val) == GST_TYPE_FRACTION);
+ gst_util_fraction_to_double(gst_value_get_fraction_numerator(val),
+ gst_value_get_fraction_denominator(val), &framerate);
+
+ frameRates.append({ framerate, framerate});
+ }
+ }
+
+ presets.append(GStreamerVideoPreset::create(size, WTFMove(frameRates)));
+ }
+
+ if (presets.isEmpty()) {
+ GST_INFO("Could not find any presets for caps: %" GST_PTR_FORMAT " just let anything go out.", caps.get());
+
+ for (auto& size : standardVideoSizes()) {
+ Vector<FrameRateRange> frameRates;
+
+ frameRates.append({ 0, G_MAXDOUBLE});
+ presets.append(GStreamerVideoPreset::create(size, WTFMove(frameRates)));
+ }
+ }
+
+ setSupportedPresets(WTFMove(presets));
+}
+
} // namespace WebCore
#endif // ENABLE(MEDIA_STREAM) && USE(LIBWEBRTC)
Modified: trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h (237047 => 237048)
--- trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.h 2018-10-11 21:30:33 UTC (rev 237048)
@@ -23,11 +23,11 @@
#if ENABLE(MEDIA_STREAM) && USE(LIBWEBRTC) && USE(GSTREAMER)
#include "GStreamerVideoCapturer.h"
-#include "RealtimeMediaSource.h"
+#include "RealtimeVideoSource.h"
namespace WebCore {
-class GStreamerVideoCaptureSource : public RealtimeMediaSource {
+class GStreamerVideoCaptureSource : public RealtimeVideoSource {
public:
static CaptureSourceOrError create(String&& deviceID, String&& hashSalt, const MediaConstraints*);
WEBCORE_EXPORT static VideoCaptureFactory& factory();
@@ -46,7 +46,10 @@
virtual ~GStreamerVideoCaptureSource();
void startProducingData() override;
void stopProducingData() override;
+ bool canResizeVideoFrames() const final { return true; }
+ void generatePresets() final;
+
mutable std::optional<RealtimeMediaSourceCapabilities> m_capabilities;
mutable std::optional<RealtimeMediaSourceSettings> m_currentSettings;
Modified: trunk/Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp (237047 => 237048)
--- trunk/Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp 2018-10-11 21:22:40 UTC (rev 237047)
+++ trunk/Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp 2018-10-11 21:30:33 UTC (rev 237048)
@@ -55,6 +55,7 @@
RealtimeMediaSourceSettings::VideoFacingMode::User, {
{ { 1280, 720 }, { { 30, 30}, { 27.5, 27.5}, { 25, 25}, { 22.5, 22.5}, { 20, 20}, { 17.5, 17.5}, { 15, 15}, { 12.5, 12.5}, { 10, 10}, { 7.5, 7.5}, { 5, 5} } },
{ { 640, 480 }, { { 30, 30}, { 27.5, 27.5}, { 25, 25}, { 22.5, 22.5}, { 20, 20}, { 17.5, 17.5}, { 15, 15}, { 12.5, 12.5}, { 10, 10}, { 7.5, 7.5}, { 5, 5} } },
+ { { 112, 112 }, { { 30, 30}, { 27.5, 27.5}, { 25, 25}, { 22.5, 22.5}, { 20, 20}, { 17.5, 17.5}, { 15, 15}, { 12.5, 12.5}, { 10, 10}, { 7.5, 7.5}, { 5, 5} } },
},
Color::black,
} },
_______________________________________________ webkit-changes mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-changes
