On Apr 20, 2016 10:45 PM, "Brion Vibber" <[email protected]> wrote: > > Over in TimedMediaHandler extension, we've had a number of cases where old > code did things that were convenient in terms of squishing read-write > operations into data getters, that got removed due to problems with long > running transactions or needing to refactor things to support > future-facing multi-DC work where we want requests to be able to more > reliably distinguish between read-only and read-write. And we sometimes > want to put some of those clever hacks back and add more. ;) > > For instance in https://gerrit.wikimedia.org/r/284368 we'd like to remove > transcode derivative files of types/resolutions that have been disabled > automatically when we come across them. But I'm a bit unsure it's safe to > do so. > > Note that we could fire off a job queue background task to do the actual > removal... But is it also safe to do that on a read-only request? > https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki > seems to indicate job queueing will be safe, but would like to confirm > that. :) >
I think this is the preferred method. My understanding is that the jobs will get shipped to the primary DC job queue. > Similarly in https://gerrit.wikimedia.org/r/#/c/284269/ we may wish to > trigger missing transcodes to run on demand, similarly. The actual re > encoding happens in a background job, but we have to fire it off, and we > have to record that we fired it off so we don't duplicate it... > > (This would require a second queue to do the high-priority state table > update and queue the actual transcoding job; we can't put them in one queue > because a backup of transcode jobs would prevent the high priority job from > running in a timely fashion.) > The job queue can do deduplication, although you would have to check if that is active while the job is running and not only while queued. Might help? > A best practices document on future-proofing for multi DC would be pretty > awesome! Maybe factor out some of the stuff from the RfC into a nice dev > doc page... > > -- brion > _______________________________________________ > Wikitech-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
