GWToolset already has several throttles in place, http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttles.2C_Limits.2C_Delays, that limit how many background uploads are picked up with each background job run, and how many total GWToolset background jobs can exist in the entire job queue. on the beta cluster the background job seemed to vary in regards to how often it ran for GWToolset varying between 7-30. that seems like enough time for additional images to get processed in-between GWToolset images.
wouldn’t it be better to throttle the application/tool that generates thumbnails so that it doesn’t try to produce too many thumbnails at once? with kind regards, dan On Apr 25, 2014, at 20:41 , Gergo Tisza <[email protected]> wrote: > On Fri, Apr 25, 2014 at 11:13 AM, Fæ <[email protected]> wrote: > With no obvious immediate fix/work-around on the table from WMF ops, I > have proposed to re-start my uploads for this project with an > effective throttle by using 2 threads (this is a setting on the first > screen of the GWToolset. In practice, having tried a run of a couple > of hundred, this means that the tool is uploading 100MB sized images > at a rate of 2 every 5 minutes. This seems to not be causing any > issues. > > The issue was not directly with the uploads; there is no thumbnail rendering > happening on upload, so GWToolset adding lots of large TIFFs quickly would > not cause problems in itself. The upload speed was problematic because that > meant GWToolset saturated pages like Special:NewFiles, and when somebody > looked at such pages, *that* triggered lots of thumbnail renderings of huge > TIFF files at the same time. If GWToolset is slowed down and lots of > miscellaneous files are uploaded between the TIFFs, those special pages won't > be problematic, but something like a gallery or category of huge TIFF files > could still be. > _______________________________________________ > Glamtools mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/glamtools _______________________________________________ Glamtools mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/glamtools
