jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/336846 )
Change subject: TMH job queue split into low and high priority
......................................................................
TMH job queue split into low and high priority
This is a first step to retooling the queue management for TMH transcode.
Lower resolutions and shorter files are now transcoded via one queue
('webVideoTranscodePrioritized' jobs) while high resolutions and long
files are transcoded in a second (the existing 'webVideoTranscode' job)
Beware that in production, this may require setting up the new queue
to send to the scalers!
Does not yet address relative prioritization after upload; each queue
can still bog down if it gets behind.
The thresholds can be configured via global vars:
$wgTmhPriorityResolutionThreshold = 480; // pixels height
$wgTmhPriorityLengthThreshold = 900; // seconds
Bug: T155098
Depends-On: I3aee49682f66cdd7c7a6b9267d8d486affc2903a
Depends-On: I83e5dab7dfb7be7a6df1a516701ef534601d530f
Change-Id: I3d834adddfa68e91bd5369246724cac2360f0de6
(cherry picked from commit 1a36fd1bdddbadb7abca5c549034a713151bb99c)
---
M TimedMediaHandler.hooks.php
M TimedMediaHandler.php
M WebVideoTranscode/WebVideoTranscode.php
M WebVideoTranscode/WebVideoTranscodeJob.php
4 files changed, 29 insertions(+), 1 deletion(-)
Approvals:
Thcipriani: Looks good to me, approved
jenkins-bot: Verified
diff --git a/TimedMediaHandler.hooks.php b/TimedMediaHandler.hooks.php
index 016ae73..1a1d46e 100644
--- a/TimedMediaHandler.hooks.php
+++ b/TimedMediaHandler.hooks.php
@@ -261,6 +261,8 @@
// Add transcode job class:
$wgJobClasses['webVideoTranscode'] = 'WebVideoTranscodeJob';
+ // Same class with different queue priority:
+ $wgJobClasses['webVideoTranscodePrioritized'] =
'WebVideoTranscodeJob';
// Transcode jobs must be explicitly requested from the job
queue:
$wgJobTypesExcludedFromDefaultQueue[] = 'webVideoTranscode';
diff --git a/TimedMediaHandler.php b/TimedMediaHandler.php
index 46c71ef..983cdb0 100644
--- a/TimedMediaHandler.php
+++ b/TimedMediaHandler.php
@@ -244,6 +244,15 @@
$wgFileExtensions = array_merge( $wgFileExtensions, $wgTmhFileExtensions );
+// Transcode resolutions higher than this will run in the low-priority queue
+// This'll give us SD transcodes as fast as possible, then do HD later.
+$wgTmhPriorityResolutionThreshold = 480;
+
+// Transcodes of files longer than this (seconds) will run in the low-priority
+// queue; defaults to 15 minutes.
+// This'll mean long videos won't flood the high-priority queue.
+$wgTmhPriorityLengthThreshold = 900;
+
// Timed Media Handler AutoLoad Classes:
$wgAutoloadClasses['TimedMediaHandler'] =
"$timedMediaDir/TimedMediaHandler_body.php";
$wgAutoloadClasses['TimedMediaHandlerHooks'] =
"$timedMediaDir/TimedMediaHandler.hooks.php";
diff --git a/WebVideoTranscode/WebVideoTranscode.php
b/WebVideoTranscode/WebVideoTranscode.php
index c1b8d1c..a0cacc3 100644
--- a/WebVideoTranscode/WebVideoTranscode.php
+++ b/WebVideoTranscode/WebVideoTranscode.php
@@ -1159,6 +1159,8 @@
* @param $transcodeKey String transcode key
*/
public static function updateJobQueue( &$file, $transcodeKey ) {
+ global $wgTmhPriorityResolutionThreshold,
$wgTmhPriorityLengthThreshold;
+
$fileName = $file->getTitle()->getDbKey();
$db = $file->repo->getMasterDB();
@@ -1188,9 +1190,19 @@
return;
}
+ // Set the priority
+ $transcodeHeight = 0;
+ $matches = [];
+ if ( preg_match( '/^(\d+)p/', $transcodeKey, $matches )
) {
+ $transcodeHeight = intval( $matches[0] );
+ }
+ $prioritized = ( $transcodeHeight <=
$wgTmhPriorityResolutionThreshold )
+ && ( $file->getLength() <=
$wgTmhPriorityLengthThreshold );
+
$job = new WebVideoTranscodeJob( $file->getTitle(), [
'transcodeMode' => 'derivative',
'transcodeKey' => $transcodeKey,
+ 'prioritized' => $prioritized
] );
try {
diff --git a/WebVideoTranscode/WebVideoTranscodeJob.php
b/WebVideoTranscode/WebVideoTranscodeJob.php
index 980a66a..8256d41 100644
--- a/WebVideoTranscode/WebVideoTranscodeJob.php
+++ b/WebVideoTranscode/WebVideoTranscodeJob.php
@@ -26,7 +26,12 @@
public $file;
public function __construct( $title, $params, $id = 0 ) {
- parent::__construct( 'webVideoTranscode', $title, $params, $id
);
+ if ( isset( $params['prioritized'] ) && $params['prioritized']
) {
+ $command = 'webVideoTranscodePrioritized';
+ } else {
+ $command = 'webVideoTranscode';
+ }
+ parent::__construct( $command, $title, $params, $id );
$this->removeDuplicates = true;
}
--
To view, visit https://gerrit.wikimedia.org/r/336846
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I3d834adddfa68e91bd5369246724cac2360f0de6
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/TimedMediaHandler
Gerrit-Branch: wmf/1.29.0-wmf.11
Gerrit-Owner: Brion VIBBER <[email protected]>
Gerrit-Reviewer: Thcipriani <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
