[webkit-changes] [248464] trunk/Source/WebCore

Fri, 09 Aug 2019 02:38:05 -0700

Title: [248464] trunk/Source/WebCore
Revision
248464
Author
[email protected]
Date
2019-08-09 02:36:45 -0700 (Fri, 09 Aug 2019)

Log Message

[GL][GStreamer] activate wrapped shared context
https://bugs.webkit.org/show_bug.cgi?id=196966

Patch by Víctor Manuel Jáquez Leal <[email protected]> on 2019-08-09
Reviewed by Žan Doberšek.

This patch consists in four parts:

1\ When the media player is instantiated, and it is intended to
render textures, it will create a wrapped object of the
application's GL context, and in order to populate the wrapped
object with the GL vtable, the context has to be current. Thus,
this patch makes current the shared WebKit application context,
and populate the wrapped GstGLContext by activating it and filling
in it. Afterwards, the wrapped context is deactivated.

2\ This patch makes GL texture use the RGBA color space, thus the
color transformation is done in GStreamer, and no further color
transformation is required in WebKit.

3\ Since it is not necessary to modify behavior if the decoder is
imxvpudecoder, its identification and label were removed.

4\ As only RGBA is used, the old color conversions when rendering
using Cairo (fallback) were changed to convert the RGBA, as in
GStreamer's format, to ARGB32, as in Cairo format -which depends
on endianness.

No new tests because there is no behavior change.

* platform/graphics/gstreamer/ImageGStreamerCairo.cpp:
(WebCore::ImageGStreamer::ImageGStreamer): Only convert GStreamer
RGBA to Cairo RGB32.
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore::MediaPlayerPrivateGStreamer::createGSTPlayBin): Removes
the IMX VPU identification.
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp:
(WebCore::MediaPlayerPrivateGStreamerBase::ensureGstGLContext):
Intializes the wrapped GL Context.
(WebCore::MediaPlayerPrivateGStreamerBase::updateTextureMapperFlags):
Removes frame's color conversion.
(WebCore::MediaPlayerPrivateGStreamerBase::createVideoSinkGL):
Instead of parsing a string, the GstCaps are created manually, and
it is set to appsink, rather than a filtered linking.
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.h:
Removes ImxVPU enumeration value.
* platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp:
Adds NoConvert option to texture copier, setting an identity
matrix.
(WebCore::VideoTextureCopierGStreamer::updateColorConversionMatrix):
* platform/graphics/gstreamer/VideoTextureCopierGStreamer.h: Adds
NoConvert enumeration value.

Modified Paths

Diff

Modified: trunk/Source/WebCore/ChangeLog (248463 => 248464)


--- trunk/Source/WebCore/ChangeLog	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/ChangeLog	2019-08-09 09:36:45 UTC (rev 248464)
@@ -1,3 +1,57 @@
+2019-08-09  Víctor Manuel Jáquez Leal  <[email protected]>
+
+        [GL][GStreamer] activate wrapped shared context
+        https://bugs.webkit.org/show_bug.cgi?id=196966
+
+        Reviewed by Žan Doberšek.
+
+        This patch consists in four parts:
+
+        1\ When the media player is instantiated, and it is intended to
+        render textures, it will create a wrapped object of the
+        application's GL context, and in order to populate the wrapped
+        object with the GL vtable, the context has to be current. Thus,
+        this patch makes current the shared WebKit application context,
+        and populate the wrapped GstGLContext by activating it and filling
+        in it. Afterwards, the wrapped context is deactivated.
+
+        2\ This patch makes GL texture use the RGBA color space, thus the
+        color transformation is done in GStreamer, and no further color
+        transformation is required in WebKit.
+
+        3\ Since it is not necessary to modify behavior if the decoder is
+        imxvpudecoder, its identification and label were removed.
+
+        4\ As only RGBA is used, the old color conversions when rendering
+        using Cairo (fallback) were changed to convert the RGBA, as in
+        GStreamer's format, to ARGB32, as in Cairo format -which depends
+        on endianness.
+
+        No new tests because there is no behavior change.
+
+        * platform/graphics/gstreamer/ImageGStreamerCairo.cpp:
+        (WebCore::ImageGStreamer::ImageGStreamer): Only convert GStreamer
+        RGBA to Cairo RGB32.
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
+        (WebCore::MediaPlayerPrivateGStreamer::createGSTPlayBin): Removes
+        the IMX VPU identification.
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp:
+        (WebCore::MediaPlayerPrivateGStreamerBase::ensureGstGLContext):
+        Intializes the wrapped GL Context.
+        (WebCore::MediaPlayerPrivateGStreamerBase::updateTextureMapperFlags):
+        Removes frame's color conversion.
+        (WebCore::MediaPlayerPrivateGStreamerBase::createVideoSinkGL):
+        Instead of parsing a string, the GstCaps are created manually, and
+        it is set to appsink, rather than a filtered linking.
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.h:
+        Removes ImxVPU enumeration value.
+        * platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp:
+        Adds NoConvert option to texture copier, setting an identity
+        matrix.
+        (WebCore::VideoTextureCopierGStreamer::updateColorConversionMatrix):
+        * platform/graphics/gstreamer/VideoTextureCopierGStreamer.h: Adds
+        NoConvert enumeration value.
+
 2019-08-09  Ryosuke Niwa  <[email protected]>
 
         REGRESSION (iOS 13): united.com web forms do not respond to taps

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/ImageGStreamerCairo.cpp (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/ImageGStreamerCairo.cpp	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/ImageGStreamerCairo.cpp	2019-08-09 09:36:45 UTC (rev 248464)
@@ -57,11 +57,7 @@
 
     RefPtr<cairo_surface_t> surface;
     cairo_format_t cairoFormat;
-#if G_BYTE_ORDER == G_LITTLE_ENDIAN
-    cairoFormat = (GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_BGRA) ? CAIRO_FORMAT_ARGB32 : CAIRO_FORMAT_RGB24;
-#else
-    cairoFormat = (GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_ARGB) ? CAIRO_FORMAT_ARGB32 : CAIRO_FORMAT_RGB24;
-#endif
+    cairoFormat = (GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_RGBA) ? CAIRO_FORMAT_ARGB32 : CAIRO_FORMAT_RGB24;
 
     // GStreamer doesn't use premultiplied alpha, but cairo does. So if the video format has an alpha component
     // we need to premultiply it before passing the data to cairo. This needs to be both using gstreamer-gl and not
@@ -75,20 +71,19 @@
 
         for (int x = 0; x < width; x++) {
             for (int y = 0; y < height; y++) {
+                unsigned short alpha = bufferData[3];
 #if G_BYTE_ORDER == G_LITTLE_ENDIAN
-                // Video frames use BGRA in little endian.
-                unsigned short alpha = bufferData[3];
-                surfacePixel[0] = (bufferData[0] * alpha + 128) / 255;
+                // Video frames use RGBA in little endian.
+                surfacePixel[0] = (bufferData[2] * alpha + 128) / 255;
                 surfacePixel[1] = (bufferData[1] * alpha + 128) / 255;
-                surfacePixel[2] = (bufferData[2] * alpha + 128) / 255;
+                surfacePixel[2] = (bufferData[0] * alpha + 128) / 255;
                 surfacePixel[3] = alpha;
 #else
-                // Video frames use ARGB in big endian.
-                unsigned short alpha = bufferData[0];
+                // Video frames use RGBA in big endian.
                 surfacePixel[0] = alpha;
-                surfacePixel[1] = (bufferData[1] * alpha + 128) / 255;
-                surfacePixel[2] = (bufferData[2] * alpha + 128) / 255;
-                surfacePixel[3] = (bufferData[3] * alpha + 128) / 255;
+                surfacePixel[1] = (bufferData[0] * alpha + 128) / 255;
+                surfacePixel[2] = (bufferData[1] * alpha + 128) / 255;
+                surfacePixel[3] = (bufferData[2] * alpha + 128) / 255;
 #endif
                 bufferData += 4;
                 surfacePixel += 4;

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp	2019-08-09 09:36:45 UTC (rev 248464)
@@ -2435,8 +2435,6 @@
         GUniquePtr<char> elementName(gst_element_get_name(element));
         if (g_str_has_prefix(elementName.get(), "v4l2"))
             player->m_videoDecoderPlatform = WebKitGstVideoDecoderPlatform::Video4Linux;
-        else if (g_str_has_prefix(elementName.get(), "imxvpudecoder"))
-            player->m_videoDecoderPlatform = WebKitGstVideoDecoderPlatform::ImxVPU;
 
 #if USE(TEXTURE_MAPPER_GL)
         player->updateTextureMapperFlags();

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp	2019-08-09 09:36:45 UTC (rev 248464)
@@ -54,19 +54,10 @@
 #endif
 
 #if USE(GSTREAMER_GL)
-#if G_BYTE_ORDER == G_LITTLE_ENDIAN
-#define GST_GL_CAPS_FORMAT "{ BGRx, BGRA }"
-#define TEXTURE_MAPPER_COLOR_CONVERT_FLAG TextureMapperGL::ShouldConvertTextureBGRAToRGBA
-#define TEXTURE_COPIER_COLOR_CONVERT_FLAG VideoTextureCopierGStreamer::ColorConversion::ConvertBGRAToRGBA
-#else
-#define GST_GL_CAPS_FORMAT "{ xRGB, ARGB }"
-#define TEXTURE_MAPPER_COLOR_CONVERT_FLAG TextureMapperGL::ShouldConvertTextureARGBToRGBA
-#define TEXTURE_COPIER_COLOR_CONVERT_FLAG VideoTextureCopierGStreamer::ColorConversion::ConvertARGBToRGBA
-#endif
+#define TEXTURE_COPIER_COLOR_CONVERT_FLAG VideoTextureCopierGStreamer::ColorConversion::NoConvert
 
 #include <gst/app/gstappsink.h>
 
-
 #include "GLContext.h"
 #if USE(GLX)
 #include "GLContextGLX.h"
@@ -475,6 +466,19 @@
     else
         m_glContext = gst_gl_context_new_wrapped(m_glDisplay.get(), reinterpret_cast<guintptr>(contextHandle), glPlatform, glAPI);
 
+    // Activate and fill the GStreamer wrapped context with the Webkit's shared one.
+    auto previousActiveContext = GLContext::current();
+    webkitContext->makeContextCurrent();
+    if (gst_gl_context_activate(m_glContext.get(), TRUE)) {
+        GUniqueOutPtr<GError> error;
+        if (!gst_gl_context_fill_info(m_glContext.get(), &error.outPtr()))
+            GST_WARNING("Failed to fill in GStreamer context: %s", error->message);
+        gst_gl_context_activate(m_glContext.get(), FALSE);
+    } else
+        GST_WARNING("Failed to activate GStreamer context %" GST_PTR_FORMAT, m_glContext.get());
+    if (previousActiveContext)
+        previousActiveContext->makeContextCurrent();
+
     return true;
 }
 #endif // USE(GSTREAMER_GL)
@@ -993,14 +997,6 @@
         m_textureMapperFlags = 0;
         break;
     }
-
-#if USE(GSTREAMER_GL)
-    // When the imxvpudecoder is used, the texture sampling of the
-    // directviv-uploaded texture returns an RGB value, so there's no need to
-    // convert it.
-    if (m_videoDecoderPlatform != WebKitGstVideoDecoderPlatform::ImxVPU)
-        m_textureMapperFlags |= TEXTURE_MAPPER_COLOR_CONVERT_FLAG;
-#endif
 }
 #endif
 
@@ -1065,6 +1061,10 @@
     GstElement* colorconvert = gst_element_factory_make("glcolorconvert", nullptr);
     GstElement* appsink = createGLAppSink();
 
+    // glsinkbin is not used because it includes glcolorconvert which only process RGBA,
+    // but in the future it would be possible to render YUV formats too:
+    // https://bugs.webkit.org/show_bug.cgi?id=132869
+
     if (!appsink || !upload || !colorconvert) {
         GST_WARNING("Failed to create GstGL elements");
         gst_object_unref(videoSink);
@@ -1082,10 +1082,11 @@
 
     gst_bin_add_many(GST_BIN(videoSink), upload, colorconvert, appsink, nullptr);
 
-    GRefPtr<GstCaps> caps = adoptGRef(gst_caps_from_string("video/x-raw(" GST_CAPS_FEATURE_MEMORY_GL_MEMORY "), format = (string) " GST_GL_CAPS_FORMAT));
+    GRefPtr<GstCaps> caps = adoptGRef(gst_caps_new_simple("video/x-raw", "format", G_TYPE_STRING, "RGBA", nullptr));
+    gst_caps_set_features(caps.get(), 0, gst_caps_features_new(GST_CAPS_FEATURE_MEMORY_GL_MEMORY, nullptr));
+    g_object_set(appsink, "caps", caps.get(), nullptr);
 
-    result &= gst_element_link_pads(upload, "src", colorconvert, "sink");
-    result &= gst_element_link_pads_filtered(colorconvert, "src", appsink, "sink", caps.get());
+    result &= gst_element_link_many(upload, colorconvert, appsink, nullptr);
 
     GRefPtr<GstPad> pad = adoptGRef(gst_element_get_static_pad(upload, "sink"));
     gst_element_add_pad(videoSink, gst_ghost_pad_new("sink", pad.get()));

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.h (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.h	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.h	2019-08-09 09:36:45 UTC (rev 248464)
@@ -307,7 +307,7 @@
     bool m_waitingForKey { false };
 #endif
 
-    enum class WebKitGstVideoDecoderPlatform { ImxVPU, Video4Linux };
+    enum class WebKitGstVideoDecoderPlatform { Video4Linux };
     Optional<WebKitGstVideoDecoderPlatform> m_videoDecoderPlatform;
 };

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp	2019-08-09 09:36:45 UTC (rev 248464)
@@ -84,6 +84,9 @@
     case ColorConversion::ConvertARGBToRGBA:
         m_colorConversionMatrix.setMatrix(0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0);
         break;
+    case ColorConversion::NoConvert:
+        m_colorConversionMatrix.makeIdentity();
+        break;
     default:
         RELEASE_ASSERT_NOT_REACHED();
     }

Modified: trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.h (248463 => 248464)


--- trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.h	2019-08-09 07:17:44 UTC (rev 248463)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.h	2019-08-09 09:36:45 UTC (rev 248464)
@@ -36,7 +36,8 @@
 public:
     enum class ColorConversion {
         ConvertBGRAToRGBA,
-        ConvertARGBToRGBA
+        ConvertARGBToRGBA,
+        NoConvert,
     };
 
     VideoTextureCopierGStreamer(ColorConversion);

_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to