jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/403180 )
Change subject: Remove Ogg Theora .ogv transcode output code paths ...................................................................... Remove Ogg Theora .ogv transcode output code paths Since these are no longer used in production, and the conversions tools are hard to maintain due to low general usage, going ahead and taking out the Ogg Theora .ogv transcode output and ffmpeg2theora code paths. Ogg Theora videos are still supported for playback and as transcode sources, which will be converted to WebM or whatever is configured as output. The WebVideoTranscode:ENC_OGV_* constants are no longer defined, so make sure they are removed from your LocalSettings.php Bug: T181591 Change-Id: Ib98662ab8f8f9fe2ef70c1f7a79320285c3cda87 --- M README M TimedMediaHandler.php M TimedMediaHandlerHooks.php M WebVideoTranscode/WebVideoTranscode.php M WebVideoTranscode/WebVideoTranscodeJob.php 5 files changed, 25 insertions(+), 379 deletions(-) Approvals: Krinkle: Looks good to me, but someone else must approve jenkins-bot: Verified Jforrester: Looks good to me, approved diff --git a/README b/README index bc43088..9736aca 100644 --- a/README +++ b/README @@ -19,6 +19,16 @@ require( "$IP/extensions/TimedMediaHandler/TimedMediaHandler.php" ); +== Updates in 2018 == + +Ogg Theora (.ogv) video output has been removed due to ongoing issues with +ffmpeg2theora and libtheora packaging. WebM is now be used as the preferred +royalty-free video output by default. Ogg files are still supported, but +videos will be transcoded to WebM. + +If your LocalSettings.php used one of the WebVideoTranscode::ENC_OGV_160P etc +constants, you may need to remove them after updating to a current version. + == Running Transcodes == Transcoding a video to another resolution or format takes a good amount which @@ -30,6 +40,7 @@ forget to su as a webserver user): php runJobs.php --type webVideoTranscode --maxjobs 1 + php runJobs.php --type webVideoTranscodePrioritized --maxjobs 1 == Kaltura HTML5 player library == @@ -41,26 +52,19 @@ http://www.html5video.org/kaltura-player/docs -== Libav == +== ffmpeg == -We use Libav for two purposes: +We use ffmpeg for two purposes: - creating still images of videos (aka thumbnails) - - transcoding WebM, H.264 videos + - transcoding between WebM, Ogg and/or H.264 videos -Wikimedia currently uses libav as shipped in Ubuntu 12.04 (libav 0.8.x). -For best experience use that or any later release from http://libav.org. +Wikimedia currently uses ffmpeg as shipped in Debian 9. +For best experience use that or any later release from http://ffmpeg.org On Ubuntu/Debian: - apt-get install libav-tools + apt-get install ffmpeg -For H.264 support: - apt-get install libav-tools libavcodec-extra-53 - -If you operating system does not include the required libav software, -you can download static builds for multiple platforms at: - http://firefogg.org/nightly/ - -You can also build libav/ffmpeg from source. +You can also build ffmpeg from source. Guide for building ffmpeg with H.264 for Ubuntu: https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide @@ -77,21 +81,6 @@ Default being `/usr/bin/avconv`. - -== ffmpeg2theora == - -We use ffmpeg2theora for extract metadata from videos, you will need a copy on -your server. For best experience, use the latest release of ffmpeg2theora. At a -minimum you need to use ffmpeg2theora 0.27. - -You can download static ffmpeg2theora builds for multiple platforms at: -http://firefogg.org/nightly/ - -Set the ffmpeg2theora binary location with: - - $wgFFmpeg2theoraLocation = '/path/to/ffmpeg2theora'; - -Default being `/usr/bin/ffmpeg2theora`. == oggThumb == diff --git a/TimedMediaHandler.php b/TimedMediaHandler.php index cfee6ef..f5ac8bb 100644 --- a/TimedMediaHandler.php +++ b/TimedMediaHandler.php @@ -84,18 +84,13 @@ // The total amout of time a transcoding shell command can take: $wgTranscodeBackgroundTimeLimit = 3600 * 8; // Maximum amount of virtual memory available to transcoding processes in KB -// 2GB avconv, ffmpeg2theora mmap resources so virtual memory needs to be high enough +// 2GB: ffmpeg/avconv mmap resources so virtual memory needs to be high enough $wgTranscodeBackgroundMemoryLimit = 2 * 1024 * 1024; // Maximum file size transcoding processes can create, in KB $wgTranscodeBackgroundSizeLimit = 3 * 1024 * 1024; // 3GB // Number of threads to use in avconv for transcoding $wgFFmpegThreads = 1; - -// The location of ffmpeg2theora (transcoding) -// Set to false to use avconv/ffmpeg to produce Ogg Theora transcodes instead; -// beware this will disable Ogg skeleton metadata generation. -$wgFFmpeg2theoraLocation = '/usr/bin/ffmpeg2theora'; // Location of oggThumb binary ( used instead of ffmpeg ) $wgOggThumbLocation = '/usr/bin/oggThumb'; @@ -160,35 +155,6 @@ // WebVideoTranscode::ENC_WEBM_2160P, /* - // Ogg Theora/Vorbis - // Optional fallback for Safari/IE/Edge with ogv.js - // Faster to encode & play back than WebM but less efficient. - - // Requires twice the bitrate for same quality as VP8, - // and JS decoder can be slow, so shift to smaller sizes. - - // Low-bitrate Ogg stream - WebVideoTranscode::ENC_OGV_160P, - - // Medium-bitrate Ogg stream - WebVideoTranscode::ENC_OGV_240P, - - // Moderate-bitrate Ogg stream - WebVideoTranscode::ENC_OGV_360P, - - // High-bitrate Ogg stream - WebVideoTranscode::ENC_OGV_480P, - - // Variable-bitrate HD Ogg stream - // for ogv.js on reasonably speedy machines - WebVideoTranscode::ENC_OGV_720P, - - // Variable-bitrate HD Ogg stream - // for ogv.js on reasonably speedy machines - WebVideoTranscode::ENC_OGV_1080P, -*/ - -/* // MP4 H.264/AAC // Primary format for the Apple/Microsoft world // @@ -236,11 +202,6 @@ // If mp4 source assets can be ingested: $wgTmhEnableMp4Uploads = false; - -// Two-pass encoding for .ogv Theora transcodes is flaky as of October 2015. -// Enable this only if testing with latest theora libraries! -// See tracking bug: https://phabricator.wikimedia.org/T115883 -$wgTmhTheoraTwoPassEncoding = false; /* CONFIGURATION ENDS HERE */ diff --git a/TimedMediaHandlerHooks.php b/TimedMediaHandlerHooks.php index 64dbf97..448f9b4 100644 --- a/TimedMediaHandlerHooks.php +++ b/TimedMediaHandlerHooks.php @@ -208,7 +208,7 @@ $wgResourceModules, $wgExcludeFromThumbnailPurge, $wgFileExtensions, $wgTmhEnableMp3Uploads, $wgTmhEnableMp4Uploads, $wgExtensionAssetsPath, $wgMwEmbedModuleConfig, $wgEnableLocalTimedText, $wgTmhFileExtensions, - $wgTmhTheoraTwoPassEncoding, $wgWikimediaJenkinsCI; + $wgWikimediaJenkinsCI; // set config for parser tests if ( isset( $wgWikimediaJenkinsCI ) && $wgWikimediaJenkinsCI === true ) { @@ -230,15 +230,6 @@ $index = array_search( 'mp4', $wgFileExtensions ); if ( $index !== false ) { array_splice( $wgFileExtensions, $index, 1 ); - } - } - - // Enable experimental 2-pass Theora encoding if enabled: - if ( $wgTmhTheoraTwoPassEncoding ) { - foreach ( WebVideoTranscode::$derivativeSettings as $key => &$settings ) { - if ( isset( $settings['videoCodec'] ) && $settings['videoCodec'] === 'theora' ) { - $settings['twopass'] = 'true'; - } } } diff --git a/WebVideoTranscode/WebVideoTranscode.php b/WebVideoTranscode/WebVideoTranscode.php index c8125ab..fa3f99c 100644 --- a/WebVideoTranscode/WebVideoTranscode.php +++ b/WebVideoTranscode/WebVideoTranscode.php @@ -25,15 +25,8 @@ * $messages['timedmedia-derivative-200_200kbs.ogv'] => 'Ogg 200'; */ - // Ogg Profiles - const ENC_OGV_160P = '160p.ogv'; - const ENC_OGV_240P = '240p.ogv'; - const ENC_OGV_360P = '360p.ogv'; - const ENC_OGV_480P = '480p.ogv'; - const ENC_OGV_720P = '720p.ogv'; - const ENC_OGV_1080P = '1080p.ogv'; - const ENC_OGV_1440P = '1440p.ogv'; - const ENC_OGV_2160P = '2160p.ogv'; + // Note: Ogg video profiles have been removed as of January 2018. + // Use WebM output for royalty-free codec output. // WebM VP8/Vorbis profiles: const ENC_WEBM_160P = '160p.webm'; @@ -83,130 +76,6 @@ * http://firefogg.org/dev/index.html */ public static $derivativeSettings = [ - self::ENC_OGV_160P => - [ - 'maxSize' => '288x160', - 'videoBitrate' => '256', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => '-1', - 'channels' => '2', - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'bufDelay' => '256', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - self::ENC_OGV_240P => - [ - 'maxSize' => '426x240', - 'videoBitrate' => '512', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => '0', - 'channels' => '2', - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'bufDelay' => '256', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - self::ENC_OGV_360P => - [ - 'maxSize' => '640x360', - 'videoBitrate' => '1024', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => '1', - 'channels' => '2', - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'bufDelay' => '256', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - self::ENC_OGV_480P => - [ - 'maxSize' => '854x480', - 'videoBitrate' => '2048', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => '2', - 'channels' => '2', - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'bufDelay' => '256', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - - self::ENC_OGV_720P => - [ - 'maxSize' => '1280x720', - 'videoBitrate' => '4096', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => 3, - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - - self::ENC_OGV_1080P => - [ - 'maxSize' => '1920x1080', - 'videoBitrate' => '8192', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => 3, - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - - self::ENC_OGV_1440P => - [ - 'maxSize' => '2560x1440', - 'videoBitrate' => '16384', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => 3, - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], - - self::ENC_OGV_2160P => - [ - 'maxSize' => '3840x2160', - 'videoBitrate' => '32768', - 'softTarget' => 'true', - 'framerate' => '60', // max to reduce "1000fps bug" problems - 'audioQuality' => 3, - 'noUpscaling' => 'true', // also caps to source frame rate - 'twopass' => 'false', // will be overridden by $wgTmhTheoraTwoPassEncoding - 'optimize' => 'true', - 'keyframeInterval' => '128', - 'videoCodec' => 'theora', - 'type' => 'video/ogg; codecs="theora, vorbis"', - ], // WebM transcode: self::ENC_WEBM_160P => diff --git a/WebVideoTranscode/WebVideoTranscodeJob.php b/WebVideoTranscode/WebVideoTranscodeJob.php index ae02e25..8952808 100644 --- a/WebVideoTranscode/WebVideoTranscodeJob.php +++ b/WebVideoTranscode/WebVideoTranscodeJob.php @@ -119,7 +119,7 @@ * @return bool success */ public function run() { - global $wgVersion, $wgFFmpeg2theoraLocation; + global $wgVersion; // get a local pointer to the file $file = $this->getFile(); @@ -195,11 +195,8 @@ // Check the codec see which encode method to call; if ( isset( $options[ 'novideo' ] ) ) { $status = $this->ffmpegEncode( $options ); - } elseif ( $options['videoCodec'] == 'theora' && $wgFFmpeg2theoraLocation !== false ) { - $status = $this->ffmpeg2TheoraEncode( $options ); } elseif ( $options['videoCodec'] == 'vp8' || $options['videoCodec'] == 'vp9' || - $options['videoCodec'] == 'h264' || - ( $options['videoCodec'] == 'theora' && $wgFFmpeg2theoraLocation === false ) + $options['videoCodec'] == 'h264' ) { // Check for twopass: if ( isset( $options['twopass'] ) ) { @@ -355,7 +352,7 @@ } /** - * Utility helper for ffmpeg and ffmpeg2theora mapping + * Utility helper for ffmpeg mapping * @param array $options * @param int $pass * @return bool|string @@ -382,8 +379,6 @@ $cmd .= $this->ffmpegAddWebmVideoOptions( $options, $pass ); } elseif ( $options['videoCodec'] == 'h264' ) { $cmd .= $this->ffmpegAddH264VideoOptions( $options, $pass ); - } elseif ( $options['videoCodec'] == 'theora' ) { - $cmd .= $this->ffmpegAddTheoraVideoOptions( $options, $pass ); } // Add size options: $cmd .= $this->ffmpegAddVideoSizeOptions( $options ); @@ -590,57 +585,6 @@ } /** - * Adds ffmpeg/avconv shell options for ogg - * - * Used only when $wgFFmpeg2theoraLocation set to false. - * Warning: does not create Ogg skeleton metadata track. - * - * @param array $options - * @param int $pass - * @return string - */ - function ffmpegAddTheoraVideoOptions( $options, $pass ) { - global $wgFFmpegThreads; - - // Get a local pointer to the file object - $file = $this->getFile(); - - $cmd = ' -threads ' . intval( $wgFFmpegThreads ); - - // Check for video quality: - if ( isset( $options['videoQuality'] ) && $options['videoQuality'] >= 0 ) { - // Map 0-10 to 63-0, higher values worse quality - $quality = 63 - intval( intval( $options['videoQuality'] ) / 10 * 63 ); - $cmd .= " -qmin " . wfEscapeShellArg( $quality ); - $cmd .= " -qmax " . wfEscapeShellArg( $quality ); - } - - // Check for video bitrate: - if ( isset( $options['videoBitrate'] ) ) { - $cmd .= " -qmin 1 -qmax 51"; - $cmd .= " -vb " . wfEscapeShellArg( $options['videoBitrate'] * 1000 ); - } - // Set the codec: - $cmd .= " -vcodec theora"; - - // Check for keyframeInterval - if ( isset( $options['keyframeInterval'] ) ) { - $cmd .= ' -g ' . wfEscapeShellArg( $options['keyframeInterval'] ); - } - if ( isset( $options['deinterlace'] ) ) { - $cmd .= ' -deinterlace'; - } - if ( isset( $options['framerate'] ) ) { - $cmd .= ' -r ' . wfEscapeShellArg( $options['framerate'] ); - } - - // Output Ogg - $cmd .= " -f ogg"; - - return $cmd; - } - - /** * @param array $options * @param int $pass * @return string @@ -681,67 +625,6 @@ $cmd .= " -acodec libvorbis "; } return $cmd; - } - - /** - * ffmpeg2Theora mapping is much simpler since it is the basis of the the firefogg API - * @param array $options - * @return bool|string - */ - function ffmpeg2TheoraEncode( $options ) { - global $wgFFmpeg2theoraLocation, $wgTranscodeBackgroundMemoryLimit; - - if ( !is_file( $this->getSourceFilePath() ) ) { - return "source file is missing, " . $this->getSourceFilePath() . ". Encoding failed."; - } - - // Set up the base command - $cmd = wfEscapeShellArg( - $wgFFmpeg2theoraLocation - ) . ' ' . wfEscapeShellArg( $this->getSourceFilePath() ); - - $file = $this->getFile(); - - if ( isset( $options['maxSize'] ) ) { - list( $width, $height ) = WebVideoTranscode::getMaxSizeTransform( $file, $options['maxSize'] ); - $options['width'] = $width; - $options['height'] = $height; - $options['aspect'] = $width . ':' . $height; - unset( $options['maxSize'] ); - } - - // Add in the encode settings - foreach ( $options as $key => $val ) { - if ( isset( self::$foggMap[$key] ) ) { - if ( is_array( self::$foggMap[$key] ) ) { - $cmd .= ' '. implode( ' ', self::$foggMap[$key] ); - } elseif ( $val == 'true' || $val === true ) { - $cmd .= ' '. self::$foggMap[$key]; - } elseif ( $val == 'false' || $val === false ) { - // ignore "false" flags - } else { - // normal get/set value - $cmd .= ' '. self::$foggMap[$key] . ' ' . wfEscapeShellArg( $val ); - } - } - } - - // Add the output target: - $outputFile = $this->getTargetEncodePath(); - $cmd .= ' -o ' . wfEscapeShellArg( $outputFile ); - - $this->output( "Running cmd: \n\n" .$cmd . "\n" ); - - $retval = 0; - $shellOutput = $this->runShellExec( $cmd, $retval ); - - // ffmpeg2theora returns 0 status on some errors, so also check for file - if ( $retval != 0 || !is_file( $outputFile ) || filesize( $outputFile ) === 0 ) { - return $cmd . - "\n\nExitcode: $retval\nMemory: $wgTranscodeBackgroundMemoryLimit\n\n" . - $shellOutput; - } - return true; } /** @@ -977,52 +860,5 @@ } return true; } - - /** - * Mapping between firefogg api and ffmpeg2theora command line - * - * This lets us share a common api between firefogg and WebVideoTranscode - * also see: http://firefogg.org/dev/index.html - */ - public static $foggMap = [ - // video - 'width' => "--width", - 'height' => "--height", - 'maxSize' => "--max_size", - 'noUpscaling' => "--no-upscaling", - 'videoQuality' => "-v", - 'videoBitrate' => "-V", - 'twopass' => "--two-pass", - 'optimize' => "--optimize", - 'framerate' => "-F", - 'aspect' => "--aspect", - 'starttime' => "--starttime", - 'endtime' => "--endtime", - 'cropTop' => "--croptop", - 'cropBottom' => "--cropbottom", - 'cropLeft' => "--cropleft", - 'cropRight' => "--cropright", - 'keyframeInterval' => "--keyint", - 'denoise' => [ "--pp", "de" ], - 'deinterlace' => "--deinterlace", - 'novideo' => [ "--novideo", "--no-skeleton" ], - 'bufDelay' => "--buf-delay", - 'softTarget' => "--soft-target", - // audio - 'audioQuality' => "-a", - 'audioBitrate' => "-A", - 'samplerate' => "-H", - 'channels' => "-c", - 'noaudio' => "--noaudio", - // metadata - 'artist' => "--artist", - 'title' => "--title", - 'date' => "--date", - 'location' => "--location", - 'organization' => "--organization", - 'copyright' => "--copyright", - 'license' => "--license", - 'contact' => "--contact" - ]; } -- To view, visit https://gerrit.wikimedia.org/r/403180 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib98662ab8f8f9fe2ef70c1f7a79320285c3cda87 Gerrit-PatchSet: 2 Gerrit-Project: mediawiki/extensions/TimedMediaHandler Gerrit-Branch: master Gerrit-Owner: Brion VIBBER <[email protected]> Gerrit-Reviewer: Brion VIBBER <[email protected]> Gerrit-Reviewer: Jforrester <[email protected]> Gerrit-Reviewer: Krinkle <[email protected]> Gerrit-Reviewer: TheDJ <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
