https://bugzilla.wikimedia.org/show_bug.cgi?id=65691
Bug ID: 65691
Summary: Limit the number or expensive thumbnails that can be
processed by the scalers in parallel
Product: MediaWiki
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: File management
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected],
[email protected], [email protected],
[email protected], [email protected],
[email protected], [email protected]
Blocks: 65217
Web browser: ---
Mobile Platform: ---
Certain files are very expensive to thumbnail (e.g. large TIFF files);
receiving several such requests can bring the scalers down. This was a
recurring issue with certain GWToolset uploads. To prevent such issues, there
should be a way to mark files as expensive and throttle the scaling of such
files.
This is tracked by the Multimedia team as
https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/623 . The
current plan is:
* have the media handler decide whether scaling the file is expensive, and pass
that flag on
* for expensive files, the pool counter should bucket by some hash of the file
name instead of the file name itself, so that there are only <range of hash
function> slots available for expensive thumbnailing operations
* the hash has to be deterministic to avoid stampeding, and the range has to be
configurable
This is not an ideal solution, since hash collisions will make it hard to
estimate the exact effect of the throttling - some of the jobs will be blocked
even when there are less jobs than available slots. It will make it easy to
limit the maximum possible load though, which is still an improvement over the
current situation.
(The alternative would be to just have a big "expensive thumbnails" bucket
without any kind of hashing - given that expensive thumbnail requests are slow
and likely to be stampeded, that would be even worse.)
--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
