Log Message
Performance: do pixel conformance and texturing in a single step. https://bugs.webkit.org/show_bug.cgi?id=178219 <rdar://problem/34937237>
Reviewed by Dean Jackson. No new tests; performance improvements should have no behavior change. Rather than asking the VTDecompressionSession to conform the output CVPixelBuffer into a pixel format compatible with OpenGL (& ES), don't constrain the output at all, and only do a conformance step if the output is not already compatible with OpenGL. This eliminates one copy (in hardware) operation. Move the TextureCacheCV object into VideoTextureCopierCV; it will be conditionally used to create the texture if the pixel buffer is compatible. Refactor copyVideoTextureToPlatformTexture(CVOpenGLTextureRef) in VideoTextureCopierCV. The new entry point, copyImageToPlatformTexture(), will attempt to use the texture cache first, and call a new common copyVideoTextureToPlatformTexture(Platform3DObject) with the result. The new copyImageToPlatformTexture() will pull planar YUV frames into two textures, and combine the two with a color transfer function when drawing to the output texture. * platform/graphics/GraphicsContext3D.h: * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm: (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture): * platform/graphics/cocoa/GraphicsContext3DCocoa.mm: (WebCore::GraphicsContext3D::texImageIOSurface2D): * platform/graphics/cocoa/WebCoreDecompressionSession.mm: (WebCore::WebCoreDecompressionSession::ensureDecompressionSessionForSample): * platform/graphics/cv/TextureCacheCV.h: * platform/graphics/cv/TextureCacheCV.mm: (WebCore::TextureCacheCV::textureFromImage): * platform/graphics/cv/VideoTextureCopierCV.cpp: (WebCore::pixelRangeFromPixelFormat): (WebCore::transferFunctionFromString): (WebCore::YCbCrToRGBMatrixForRangeAndTransferFunction): (WebCore::VideoTextureCopierCV::~VideoTextureCopierCV): (WebCore::VideoTextureCopierCV::initializeUVContextObjects): (WebCore::VideoTextureCopierCV::copyImageToPlatformTexture): (WebCore::VideoTextureCopierCV::copyVideoTextureToPlatformTexture): * platform/graphics/cv/VideoTextureCopierCV.h:
Modified Paths
- trunk/Source/WebCore/ChangeLog
- trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp
- trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.h
- trunk/Source/WebCore/platform/graphics/GraphicsContext3D.h
- trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm
- trunk/Source/WebCore/platform/graphics/cocoa/GraphicsContext3DCocoa.mm
- trunk/Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm
- trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.h
- trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.mm
- trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.cpp
- trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.h
Diff
Modified: trunk/Source/WebCore/ChangeLog (223279 => 223280)
--- trunk/Source/WebCore/ChangeLog 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/ChangeLog 2017-10-13 16:32:38 UTC (rev 223280)
@@ -1,3 +1,48 @@
+2017-10-13 Jer Noble <jer.no...@apple.com>
+
+ Performance: do pixel conformance and texturing in a single step.
+ https://bugs.webkit.org/show_bug.cgi?id=178219
+ <rdar://problem/34937237>
+
+ Reviewed by Dean Jackson.
+
+ No new tests; performance improvements should have no behavior change.
+
+ Rather than asking the VTDecompressionSession to conform the output CVPixelBuffer into a
+ pixel format compatible with OpenGL (& ES), don't constrain the output at all, and only do a
+ conformance step if the output is not already compatible with OpenGL. This eliminates one
+ copy (in hardware) operation.
+
+ Move the TextureCacheCV object into VideoTextureCopierCV; it will be conditionally used to
+ create the texture if the pixel buffer is compatible.
+
+ Refactor copyVideoTextureToPlatformTexture(CVOpenGLTextureRef) in VideoTextureCopierCV. The
+ new entry point, copyImageToPlatformTexture(), will attempt to use the texture cache first,
+ and call a new common copyVideoTextureToPlatformTexture(Platform3DObject) with the result.
+
+ The new copyImageToPlatformTexture() will pull planar YUV frames into two textures, and combine
+ the two with a color transfer function when drawing to the output texture.
+
+ * platform/graphics/GraphicsContext3D.h:
+ * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
+ (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture):
+ * platform/graphics/cocoa/GraphicsContext3DCocoa.mm:
+ (WebCore::GraphicsContext3D::texImageIOSurface2D):
+ * platform/graphics/cocoa/WebCoreDecompressionSession.mm:
+ (WebCore::WebCoreDecompressionSession::ensureDecompressionSessionForSample):
+ * platform/graphics/cv/TextureCacheCV.h:
+ * platform/graphics/cv/TextureCacheCV.mm:
+ (WebCore::TextureCacheCV::textureFromImage):
+ * platform/graphics/cv/VideoTextureCopierCV.cpp:
+ (WebCore::pixelRangeFromPixelFormat):
+ (WebCore::transferFunctionFromString):
+ (WebCore::YCbCrToRGBMatrixForRangeAndTransferFunction):
+ (WebCore::VideoTextureCopierCV::~VideoTextureCopierCV):
+ (WebCore::VideoTextureCopierCV::initializeUVContextObjects):
+ (WebCore::VideoTextureCopierCV::copyImageToPlatformTexture):
+ (WebCore::VideoTextureCopierCV::copyVideoTextureToPlatformTexture):
+ * platform/graphics/cv/VideoTextureCopierCV.h:
+
2017-10-13 Romain Bellessort <romain.belless...@crf.canon.fr>
[Readable Streams API] Align queue with spec for ReadableStreamDefaultController
Modified: trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp (223279 => 223280)
--- trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp 2017-10-13 16:32:38 UTC (rev 223280)
@@ -52,6 +52,13 @@
SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelFormatOpenGLESCompatibility, CFStringRef)
SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferIOSurfacePropertiesKey, CFStringRef)
SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferPoolMinimumBufferCountKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrixKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_709_2, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_601_4, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_SMPTE_240M_1995, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_DCI_P3, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_P3_D65, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_2020, CFStringRef)
#if PLATFORM(IOS)
SOFT_LINK_FUNCTION_FOR_SOURCE(WebCore, CoreVideo, CVOpenGLESTextureCacheCreate, CVReturn, (CFAllocatorRef allocator, CFDictionaryRef cacheAttributes, CVEAGLContext eaglContext, CFDictionaryRef textureAttributes, CVOpenGLESTextureCacheRef* cacheOut), (allocator, cacheAttributes, eaglContext, textureAttributes, cacheOut))
Modified: trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.h (223279 => 223280)
--- trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.h 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/cocoa/CoreVideoSoftLink.h 2017-10-13 16:32:38 UTC (rev 223280)
@@ -70,6 +70,20 @@
#define kCVPixelBufferIOSurfacePropertiesKey get_CoreVideo_kCVPixelBufferIOSurfacePropertiesKey()
SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferPoolMinimumBufferCountKey, CFStringRef)
#define kCVPixelBufferPoolMinimumBufferCountKey get_CoreVideo_kCVPixelBufferPoolMinimumBufferCountKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrixKey, CFStringRef)
+#define kCVImageBufferYCbCrMatrixKey get_CoreVideo_kCVImageBufferYCbCrMatrixKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_709_2, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_709_2 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_709_2()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_601_4, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_601_4 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_601_4()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_SMPTE_240M_1995, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_SMPTE_240M_1995 get_CoreVideo_kCVImageBufferYCbCrMatrix_SMPTE_240M_1995()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_DCI_P3, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_DCI_P3 get_CoreVideo_kCVImageBufferYCbCrMatrix_DCI_P3()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_P3_D65, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_P3_D65 get_CoreVideo_kCVImageBufferYCbCrMatrix_P3_D65()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_2020, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_2020 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_2020()
#if PLATFORM(IOS)
SOFT_LINK_FUNCTION_FOR_HEADER(WebCore, CoreVideo, CVOpenGLESTextureCacheCreate, CVReturn, (CFAllocatorRef allocator, CFDictionaryRef cacheAttributes, CVEAGLContext eaglContext, CFDictionaryRef textureAttributes, CVOpenGLESTextureCacheRef* cacheOut), (allocator, cacheAttributes, eaglContext, textureAttributes, cacheOut))
Modified: trunk/Source/WebCore/platform/graphics/GraphicsContext3D.h (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/GraphicsContext3D.h 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/GraphicsContext3D.h 2017-10-13 16:32:38 UTC (rev 223280)
@@ -59,6 +59,7 @@
#include <wtf/RetainPtr.h>
OBJC_CLASS CALayer;
OBJC_CLASS WebGLLayer;
+typedef struct __IOSurface* IOSurfaceRef;
#elif PLATFORM(GTK) || PLATFORM(WIN_CAIRO) || PLATFORM(WPE)
typedef unsigned int GLuint;
#endif
@@ -1149,6 +1150,7 @@
#if PLATFORM(COCOA)
void endPaint();
+ bool texImageIOSurface2D(GC3Denum target, GC3Denum internalFormat, GC3Dsizei width, GC3Dsizei height, GC3Denum format, GC3Denum type, IOSurfaceRef, GC3Duint plane);
#endif
#if PLATFORM(MAC)
Modified: trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm 2017-10-13 16:32:38 UTC (rev 223280)
@@ -608,14 +608,6 @@
if (updateLastPixelBuffer()) {
if (!m_lastPixelBuffer)
return false;
-
- if (!m_textureCache) {
- m_textureCache = TextureCacheCV::create(*context);
- if (!m_textureCache)
- return false;
- }
-
- m_lastTexture = m_textureCache->textureFromImage(m_lastPixelBuffer.get(), outputTarget, level, internalFormat, format, type);
}
size_t width = CVPixelBufferGetWidth(m_lastPixelBuffer.get());
@@ -624,7 +616,7 @@
if (!m_videoTextureCopier)
m_videoTextureCopier = std::make_unique<VideoTextureCopierCV>(*context);
- return m_videoTextureCopier->copyVideoTextureToPlatformTexture(m_lastTexture.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+ return m_videoTextureCopier->copyImageToPlatformTexture(m_lastPixelBuffer.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
}
bool MediaPlayerPrivateMediaSourceAVFObjC::hasAvailableVideoFrame() const
Modified: trunk/Source/WebCore/platform/graphics/cocoa/GraphicsContext3DCocoa.mm (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cocoa/GraphicsContext3DCocoa.mm 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cocoa/GraphicsContext3DCocoa.mm 2017-10-13 16:32:38 UTC (rev 223280)
@@ -50,6 +50,7 @@
#if PLATFORM(IOS)
#import <OpenGLES/EAGL.h>
#import <OpenGLES/EAGLDrawable.h>
+#import <OpenGLES/EAGLIOSurface.h>
#import <OpenGLES/ES2/glext.h>
#import <QuartzCore/QuartzCore.h>
#import <pal/spi/ios/OpenGLESSPI.h>
@@ -672,7 +673,18 @@
#endif
}
+bool GraphicsContext3D::texImageIOSurface2D(GC3Denum target, GC3Denum internalFormat, GC3Dsizei width, GC3Dsizei height, GC3Denum format, GC3Denum type, IOSurfaceRef surface, GC3Duint plane)
+{
#if PLATFORM(MAC)
+ return kCGLNoError == CGLTexImageIOSurface2D(platformGraphicsContext3D(), target, internalFormat, width, height, format, type, surface, plane);
+#elif PLATFORM(IOS) && !PLATFORM(IOS_SIMULATOR)
+ return [platformGraphicsContext3D() texImageIOSurface:surface target:target internalFormat:internalFormat width:width height:height format:format type:type plane:plane];
+#else
+ return false;
+#endif
+}
+
+#if PLATFORM(MAC)
void GraphicsContext3D::allocateIOSurfaceBackingStore(IntSize size)
{
LOG(WebGL, "GraphicsContext3D::allocateIOSurfaceBackingStore at %d x %d. (%p)", size.width(), size.height(), this);
Modified: trunk/Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm 2017-10-13 16:32:38 UTC (rev 223280)
@@ -219,11 +219,7 @@
NSDictionary *attributes;
if (m_mode == OpenGL) {
-#if PLATFORM(IOS)
- attributes = @{(NSString *)kCVPixelBufferIOSurfaceOpenGLESFBOCompatibilityKey: @YES};
-#else
- attributes = @{(NSString *)kCVPixelBufferIOSurfaceOpenGLFBOCompatibilityKey: @YES};
-#endif
+ attributes = nil;
} else {
ASSERT(m_mode == RGB);
attributes = @{(NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA)};
Modified: trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.h (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.h 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.h 2017-10-13 16:32:38 UTC (rev 223280)
@@ -31,7 +31,8 @@
#include <wtf/RetainPtr.h>
#include <wtf/WeakPtr.h>
-typedef struct __CVBuffer* CVImageBufferRef;
+typedef struct __CVBuffer* CVImageBufferRef;
+typedef CVImageBufferRef CVPixelBufferRef;
typedef CVImageBufferRef CVOpenGLTextureRef;
typedef CVImageBufferRef CVOpenGLESTextureRef;
typedef struct __CVOpenGLTextureCache *CVOpenGLTextureCacheRef;
@@ -55,7 +56,7 @@
TextureCacheCV(GraphicsContext3D&, RetainPtr<TextureCacheType>&&);
- RetainPtr<TextureType> textureFromImage(CVImageBufferRef, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type);
+ RetainPtr<TextureType> textureFromImage(CVPixelBufferRef, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type);
GraphicsContext3D& context() { return m_context.get(); }
private:
Modified: trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.mm (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.mm 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cv/TextureCacheCV.mm 2017-10-13 16:32:38 UTC (rev 223280)
@@ -53,7 +53,7 @@
{
}
-RetainPtr<TextureCacheCV::TextureType> TextureCacheCV::textureFromImage(CVImageBufferRef image, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type)
+RetainPtr<TextureCacheCV::TextureType> TextureCacheCV::textureFromImage(CVPixelBufferRef image, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type)
{
TextureType bareVideoTexture = nullptr;
#if PLATFORM(IOS)
Modified: trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.cpp (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.cpp 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.cpp 2017-10-13 16:32:38 UTC (rev 223280)
@@ -26,7 +26,9 @@
#include "config.h"
#include "VideoTextureCopierCV.h"
+#include "FourCC.h"
#include "Logging.h"
+#include "TextureCacheCV.h"
#include <wtf/NeverDestroyed.h>
#include <wtf/text/StringBuilder.h>
@@ -38,6 +40,124 @@
namespace WebCore {
+enum class PixelRange {
+ Unknown,
+ Video,
+ Full,
+};
+
+enum class TransferFunction {
+ Unknown,
+ kITU_R_709_2,
+ kITU_R_601_4,
+ kSMPTE_240M_1995,
+ kDCI_P3,
+ kP3_D65,
+ kITU_R_2020,
+};
+
+#if PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 101300
+enum {
+ kCVPixelFormatType_ARGB2101010LEPacked = 'l10r',
+ kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange = 'x420',
+ kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange = 'x422',
+ kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange = 'x444',
+ kCVPixelFormatType_420YpCbCr10BiPlanarFullRange = 'xf20',
+ kCVPixelFormatType_422YpCbCr10BiPlanarFullRange = 'xf22',
+ kCVPixelFormatType_444YpCbCr10BiPlanarFullRange = 'xf44',
+};
+#endif
+
+static PixelRange pixelRangeFromPixelFormat(OSType pixelFormat)
+{
+ switch (pixelFormat) {
+ case kCVPixelFormatType_4444AYpCbCr8:
+ case kCVPixelFormatType_4444AYpCbCr16:
+ case kCVPixelFormatType_422YpCbCr_4A_8BiPlanar:
+ case kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange:
+ case kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange:
+ case kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange:
+ case kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange:
+ return PixelRange::Video;
+ case kCVPixelFormatType_420YpCbCr8PlanarFullRange:
+ case kCVPixelFormatType_420YpCbCr8BiPlanarFullRange:
+ case kCVPixelFormatType_422YpCbCr8FullRange:
+ case kCVPixelFormatType_ARGB2101010LEPacked:
+ case kCVPixelFormatType_420YpCbCr10BiPlanarFullRange:
+ case kCVPixelFormatType_422YpCbCr10BiPlanarFullRange:
+ case kCVPixelFormatType_444YpCbCr10BiPlanarFullRange:
+ return PixelRange::Full;
+ default:
+ return PixelRange::Unknown;
+ }
+}
+
+static TransferFunction transferFunctionFromString(CFStringRef string)
+{
+ if (string == kCVImageBufferYCbCrMatrix_ITU_R_709_2)
+ return TransferFunction::kITU_R_709_2;
+ if (string == kCVImageBufferYCbCrMatrix_ITU_R_601_4)
+ return TransferFunction::kITU_R_601_4;
+ if (string == kCVImageBufferYCbCrMatrix_SMPTE_240M_1995)
+ return TransferFunction::kSMPTE_240M_1995;
+ if (string == kCVImageBufferYCbCrMatrix_DCI_P3)
+ return TransferFunction::kDCI_P3;
+ if (string == kCVImageBufferYCbCrMatrix_P3_D65)
+ return TransferFunction::kP3_D65;
+ if (string == kCVImageBufferYCbCrMatrix_ITU_R_2020)
+ return TransferFunction::kITU_R_2020;
+ return TransferFunction::Unknown;
+}
+
+static const Vector<GLfloat> YCbCrToRGBMatrixForRangeAndTransferFunction(PixelRange range, TransferFunction transferFunction)
+{
+ using MapKey = std::pair<PixelRange, TransferFunction>;
+ using MatrixMap = std::map<MapKey, Vector<GLfloat>>;
+
+ static NeverDestroyed<MatrixMap> matrices;
+ static dispatch_once_t onceToken;
+ dispatch_once(&onceToken, ^{
+ // Matrices are derived from the components in the ITU R.601 rev 4 specification
+ // https://www.itu.int/rec/R-REC-BT.601
+ matrices.get().emplace(MapKey(PixelRange::Video, TransferFunction::kITU_R_601_4), Vector<GLfloat>({
+ 1.164383562f, 0.0f, 1.596026786f,
+ 1.164383562f, -0.3917622901f, -0.8129676472f,
+ 1.164383562f, 2.017232143f, 0.0f,
+ }));
+ matrices.get().emplace(MapKey({PixelRange::Full, TransferFunction::kITU_R_601_4}), Vector<GLfloat>({
+ 1.000000000f, 0.0f, 1.4075196850f,
+ 1.000000000f, -0.3454911535f, -0.7169478464f,
+ 1.000000000f, 1.7789763780f, 0.0f,
+ }));
+ // Matrices are derived from the components in the ITU R.709 rev 2 specification
+ // https://www.itu.int/rec/R-REC-BT.709-2-199510-S
+ matrices.get().emplace(MapKey({PixelRange::Video, TransferFunction::kITU_R_709_2}), Vector<GLfloat>({
+ 1.164383562f, 0.0f, 1.792741071f,
+ 1.164383562f, -0.2132486143f, -0.5329093286f,
+ 1.164383562f, 2.112401786f, 0.0f,
+ }));
+ matrices.get().emplace(MapKey({PixelRange::Full, TransferFunction::kITU_R_709_2}), Vector<GLfloat>({
+ 1.000000000f, 0.0f, 1.5810000000f,
+ 1.000000000f, -0.1880617701f, -0.4699672819f,
+ 1.000000000f, 1.8629055118f, 0.0f,
+ }));
+ });
+
+ // We should never be asked to handle a Pixel Format whose range value is unknown.
+ ASSERT(range != PixelRange::Unknown);
+ if (range == PixelRange::Unknown)
+ range = PixelRange::Full;
+
+ auto iterator = matrices.get().find({range, transferFunction});
+
+ // Assume unknown transfer functions are r.601:
+ if (iterator == matrices.get().end())
+ iterator = matrices.get().find({range, TransferFunction::kITU_R_601_4});
+
+ ASSERT(iterator != matrices.get().end());
+ return iterator->second;
+}
+
VideoTextureCopierCV::VideoTextureCopierCV(GraphicsContext3D& context)
: m_context(context)
, m_framebuffer(context.createFramebuffer())
@@ -50,6 +170,10 @@
m_context->deleteProgram(m_vertexBuffer);
if (m_program)
m_context->deleteProgram(m_program);
+ if (m_yuvVertexBuffer)
+ m_context->deleteProgram(m_yuvVertexBuffer);
+ if (m_yuvProgram)
+ m_context->deleteProgram(m_yuvProgram);
m_context->deleteFramebuffer(m_framebuffer);
}
@@ -258,22 +382,234 @@
return true;
}
-bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVideoTexture, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+bool VideoTextureCopierCV::initializeUVContextObjects()
{
- if (!inputVideoTexture)
+ String vertexShaderSource = ASCIILiteral(
+ "attribute vec2 a_position;\n"
+ "uniform vec2 u_yTextureSize;\n"
+ "uniform vec2 u_uvTextureSize;\n"
+ "uniform int u_flipY;\n"
+ "varying vec2 v_yTextureCoordinate;\n"
+ "varying vec2 v_uvTextureCoordinate;\n"
+ "void main() {\n"
+ " gl_Position = vec4(a_position, 0, 1.0);\n"
+ " if (u_flipY == 1) {\n"
+ " gl_Position.y = -gl_Position.y;\n"
+ " }\n"
+ " vec2 normalizedPosition = a_position * .5 + .5;\n"
+#if PLATFORM(IOS)
+ " v_yTextureCoordinate = normalizedPosition;\n"
+ " v_uvTextureCoordinate = normalizedPosition;\n"
+#else
+ " v_yTextureCoordinate = normalizedPosition * u_yTextureSize;\n"
+ " v_uvTextureCoordinate = normalizedPosition * u_uvTextureSize;\n"
+#endif
+ "}\n"
+ );
+
+ Platform3DObject vertexShader = m_context->createShader(GraphicsContext3D::VERTEX_SHADER);
+ m_context->shaderSource(vertexShader, vertexShaderSource);
+ m_context->compileShaderDirect(vertexShader);
+
+ GC3Dint status = 0;
+ m_context->getShaderiv(vertexShader, GraphicsContext3D::COMPILE_STATUS, &status);
+ if (!status) {
+ LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Vertex shader failed to compile.", this);
+ m_context->deleteShader(vertexShader);
return false;
+ }
+ String fragmentShaderSource = ASCIILiteral(
+#if PLATFORM(IOS)
+ "precision mediump float;\n"
+ "#define SAMPLERTYPE sampler2D\n"
+ "#define TEXTUREFUNC texture2D\n"
+#else
+ "#define SAMPLERTYPE sampler2DRect\n"
+ "#define TEXTUREFUNC texture2DRect\n"
+#endif
+ "uniform SAMPLERTYPE u_yTexture;\n"
+ "uniform SAMPLERTYPE u_uvTexture;\n"
+ "uniform mat3 u_colorMatrix;\n"
+ "varying vec2 v_yTextureCoordinate;\n"
+ "varying vec2 v_uvTextureCoordinate;\n"
+ "void main() {\n"
+ " vec3 yuv;\n"
+ " yuv.r = TEXTUREFUNC(u_yTexture, v_yTextureCoordinate).r;\n"
+ " yuv.gb = TEXTUREFUNC(u_uvTexture, v_uvTextureCoordinate).rg - vec2(0.5, 0.5);\n"
+ " gl_FragColor = vec4(yuv * u_colorMatrix, 1);\n"
+ "}\n"
+ );
+
+ Platform3DObject fragmentShader = m_context->createShader(GraphicsContext3D::FRAGMENT_SHADER);
+ m_context->shaderSource(fragmentShader, fragmentShaderSource);
+ m_context->compileShaderDirect(fragmentShader);
+
+ m_context->getShaderiv(fragmentShader, GraphicsContext3D::COMPILE_STATUS, &status);
+ if (!status) {
+ LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Fragment shader failed to compile.", this);
+ m_context->deleteShader(vertexShader);
+ m_context->deleteShader(fragmentShader);
+ return false;
+ }
+
+ m_yuvProgram = m_context->createProgram();
+ m_context->attachShader(m_yuvProgram, vertexShader);
+ m_context->attachShader(m_yuvProgram, fragmentShader);
+ m_context->linkProgram(m_yuvProgram);
+
+ m_context->getProgramiv(m_yuvProgram, GraphicsContext3D::LINK_STATUS, &status);
+ if (!status) {
+ LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Program failed to link.", this);
+ m_context->deleteShader(vertexShader);
+ m_context->deleteShader(fragmentShader);
+ m_context->deleteProgram(m_yuvProgram);
+ m_yuvProgram = 0;
+ return false;
+ }
+
+ m_yTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_yTexture"));
+ m_uvTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_uvTexture"));
+ m_colorMatrixUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_colorMatrix"));
+ m_yuvFlipYUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_flipY"));
+ m_yTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_yTextureSize"));
+ m_uvTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_uvTextureSize"));
+ m_yuvPositionAttributeLocation = m_context->getAttribLocationDirect(m_yuvProgram, ASCIILiteral("a_position"));
+
+ m_context->detachShader(m_yuvProgram, vertexShader);
+ m_context->detachShader(m_yuvProgram, fragmentShader);
+ m_context->deleteShader(vertexShader);
+ m_context->deleteShader(fragmentShader);
+
+ m_yuvVertexBuffer = m_context->createBuffer();
+ float vertices[12] = { -1, -1, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1 };
+
+ m_context->bindBuffer(GraphicsContext3D::ARRAY_BUFFER, m_yuvVertexBuffer);
+ m_context->bufferData(GraphicsContext3D::ARRAY_BUFFER, sizeof(vertices), vertices, GraphicsContext3D::STATIC_DRAW);
+
+ return true;
+}
+
+bool VideoTextureCopierCV::copyImageToPlatformTexture(CVPixelBufferRef image, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
+ if (!m_textureCache) {
+ m_textureCache = TextureCacheCV::create(m_context);
+ if (!m_textureCache)
+ return false;
+ }
+
+ if (auto texture = m_textureCache->textureFromImage(image, outputTarget, level, internalFormat, format, type))
+ return copyVideoTextureToPlatformTexture(texture.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+
+ // FIXME: This currently only supports '420v' and '420f' pixel formats. Investigate supporting more pixel formats.
+ OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);
+ if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange && pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
+ LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Asked to copy an unsupported pixel format ('%s').", this, FourCC(pixelFormat).toString().utf8().data());
+ return false;
+ }
+
+ IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);
+ if (!surface)
+ return false;
+
GC3DStateSaver stateSaver(m_context.get());
- if (!m_program) {
- if (!initializeContextObjects()) {
+ if (!m_yuvProgram) {
+ if (!initializeUVContextObjects()) {
LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
return false;
}
}
- stateSaver.saveVertexAttribState(m_positionAttributeLocation);
+ stateSaver.saveVertexAttribState(m_yuvPositionAttributeLocation);
+ m_context->bindFramebuffer(GraphicsContext3D::FRAMEBUFFER, m_framebuffer);
+
+ // Allocate memory for the output texture.
+ m_context->bindTexture(GraphicsContext3D::TEXTURE_2D, outputTexture);
+ m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+ m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+ m_context->texImage2DDirect(GraphicsContext3D::TEXTURE_2D, level, internalFormat, width, height, 0, format, type, nullptr);
+
+ m_context->framebufferTexture2D(GraphicsContext3D::FRAMEBUFFER, GraphicsContext3D::COLOR_ATTACHMENT0, GraphicsContext3D::TEXTURE_2D, outputTexture, level);
+ GC3Denum status = m_context->checkFramebufferStatus(GraphicsContext3D::FRAMEBUFFER);
+ if (status != GraphicsContext3D::FRAMEBUFFER_COMPLETE) {
+ LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to create framebuffer for outputTexture.", this);
+ return false;
+ }
+
+ m_context->useProgram(m_yuvProgram);
+ m_context->viewport(0, 0, width, height);
+
+ // Bind and set up the textures for the video source.
+ auto yPlaneWidth = IOSurfaceGetWidthOfPlane(surface, 0);
+ auto yPlaneHeight = IOSurfaceGetHeightOfPlane(surface, 0);
+ auto uvPlaneWidth = IOSurfaceGetWidthOfPlane(surface, 1);
+ auto uvPlaneHeight = IOSurfaceGetHeightOfPlane(surface, 1);
+
+#if PLATFORM(IOS)
+ GC3Denum videoTextureTarget = GraphicsContext3D::TEXTURE_2D;
+#else
+ GC3Denum videoTextureTarget = GL_TEXTURE_RECTANGLE_ARB;
+#endif
+ auto uvTexture = m_context->createTexture();
+ m_context->activeTexture(GraphicsContext3D::TEXTURE1);
+ m_context->bindTexture(videoTextureTarget, uvTexture);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+ if (!m_context->texImageIOSurface2D(videoTextureTarget, GL_RG, uvPlaneWidth, uvPlaneHeight, GL_RG, GL_UNSIGNED_BYTE, surface, 1)) {
+ m_context->deleteTexture(uvTexture);
+ return false;
+ }
+
+ auto yTexture = m_context->createTexture();
+ m_context->activeTexture(GraphicsContext3D::TEXTURE0);
+ m_context->bindTexture(videoTextureTarget, yTexture);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+ m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+ if (!m_context->texImageIOSurface2D(videoTextureTarget, GL_LUMINANCE, yPlaneWidth, yPlaneHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, surface, 0)) {
+ m_context->deleteTexture(yTexture);
+ m_context->deleteTexture(uvTexture);
+ return false;
+ }
+
+ // Configure the drawing parameters.
+ m_context->uniform1i(m_yTextureUniformLocation, 0);
+ m_context->uniform1i(m_uvTextureUniformLocation, 1);
+ m_context->uniform1i(m_yuvFlipYUniformLocation, flipY);
+ m_context->uniform2f(m_yTextureSizeUniformLocation, yPlaneWidth, yPlaneHeight);
+ m_context->uniform2f(m_uvTextureSizeUniformLocation, uvPlaneWidth, uvPlaneHeight);
+
+ auto range = pixelRangeFromPixelFormat(pixelFormat);
+ auto transferFunction = transferFunctionFromString((CFStringRef)CVBufferGetAttachment(image, kCVImageBufferYCbCrMatrixKey, nil));
+ auto& colorMatrix = YCbCrToRGBMatrixForRangeAndTransferFunction(range, transferFunction);
+ m_context->uniformMatrix3fv(m_colorMatrixUniformLocation, 1, GL_FALSE, colorMatrix.data());
+
+ // Do the actual drawing.
+ m_context->bindBuffer(GraphicsContext3D::ARRAY_BUFFER, m_yuvVertexBuffer);
+ m_context->enableVertexAttribArray(m_yuvPositionAttributeLocation);
+ m_context->vertexAttribPointer(m_yuvPositionAttributeLocation, 2, GraphicsContext3D::FLOAT, false, 0, 0);
+ m_context->drawArrays(GraphicsContext3D::TRIANGLES, 0, 6);
+
+ // Clean-up.
+ m_context->deleteTexture(yTexture);
+ m_context->deleteTexture(uvTexture);
+ m_context->bindTexture(videoTextureTarget, 0);
+
+ return true;
+}
+
+bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVideoTexture, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
+ if (!inputVideoTexture)
+ return false;
+
GLfloat lowerLeft[2] = { 0, 0 };
GLfloat lowerRight[2] = { 0, 0 };
GLfloat upperRight[2] = { 0, 0 };
@@ -288,8 +624,27 @@
CVOpenGLTextureGetCleanTexCoords(inputVideoTexture, lowerLeft, lowerRight, upperRight, upperLeft);
#endif
+ if (lowerLeft[1] < upperRight[1])
+ flipY = !flipY;
+
+ return copyVideoTextureToPlatformTexture(videoTextureName, videoTextureTarget, width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+}
+
+bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(Platform3DObject videoTextureName, GC3Denum videoTextureTarget, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - internalFormat: %s, format: %s, type: %s flipY: %s, premultiplyAlpha: %s", this, enumToStringMap()[internalFormat], enumToStringMap()[format], enumToStringMap()[type], flipY ? "true" : "false", premultiplyAlpha ? "true" : "false");
+ GC3DStateSaver stateSaver(m_context.get());
+
+ if (!m_program) {
+ if (!initializeContextObjects()) {
+ LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
+ return false;
+ }
+ }
+
+ stateSaver.saveVertexAttribState(m_positionAttributeLocation);
+
m_context->bindFramebuffer(GraphicsContext3D::FRAMEBUFFER, m_framebuffer);
// Allocate memory for the output texture.
@@ -326,9 +681,6 @@
m_context->uniform2f(m_textureDimensionsUniformLocation, width, height);
#endif
- if (lowerLeft[1] < upperRight[1])
- flipY = !flipY;
-
m_context->uniform1i(m_flipYUniformLocation, flipY);
m_context->uniform1i(m_premultiplyUniformLocation, premultiplyAlpha);
Modified: trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.h (223279 => 223280)
--- trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.h 2017-10-13 09:22:48 UTC (rev 223279)
+++ trunk/Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.h 2017-10-13 16:32:38 UTC (rev 223280)
@@ -29,11 +29,14 @@
#import "GraphicsContext3D.h"
typedef struct __CVBuffer* CVImageBufferRef;
+typedef struct __CVBuffer* CVPixelBufferRef;
typedef CVImageBufferRef CVOpenGLTextureRef;
typedef CVImageBufferRef CVOpenGLESTextureRef;
namespace WebCore {
+class TextureCacheCV;
+
class VideoTextureCopierCV {
public:
VideoTextureCopierCV(GraphicsContext3D&);
@@ -45,11 +48,14 @@
typedef CVOpenGLTextureRef TextureType;
#endif
+ bool copyImageToPlatformTexture(CVPixelBufferRef, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
bool copyVideoTextureToPlatformTexture(TextureType, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
GraphicsContext3D& context() { return m_context.get(); }
private:
+ bool copyVideoTextureToPlatformTexture(Platform3DObject inputTexture, GC3Denum inputTarget, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
+
class GC3DStateSaver {
public:
GC3DStateSaver(GraphicsContext3D&);
@@ -75,8 +81,10 @@
};
bool initializeContextObjects();
+ bool initializeUVContextObjects();
Ref<GraphicsContext3D> m_context;
+ std::unique_ptr<TextureCacheCV> m_textureCache;
Platform3DObject m_framebuffer { 0 };
Platform3DObject m_program { 0 };
Platform3DObject m_vertexBuffer { 0 };
@@ -85,6 +93,15 @@
GC3Dint m_flipYUniformLocation { -1 };
GC3Dint m_premultiplyUniformLocation { -1 };
GC3Dint m_positionAttributeLocation { -1 };
+ Platform3DObject m_yuvProgram { 0 };
+ Platform3DObject m_yuvVertexBuffer { 0 };
+ GC3Dint m_yTextureUniformLocation { -1 };
+ GC3Dint m_uvTextureUniformLocation { -1 };
+ GC3Dint m_yuvFlipYUniformLocation { -1 };
+ GC3Dint m_colorMatrixUniformLocation { -1 };
+ GC3Dint m_yuvPositionAttributeLocation { -1 };
+ GC3Dint m_yTextureSizeUniformLocation { -1 };
+ GC3Dint m_uvTextureSizeUniformLocation { -1 };
};
}
_______________________________________________ webkit-changes mailing list webkit-changes@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-changes
