Nice work, all! On Tue, Oct 5, 2021 at 5:47 PM Kunal Mehta <[email protected]> wrote:
> > Hi everyone, > > tl;dr: External shell outs are now run via Shellbox. Any deployed code > needs to use Shellbox/BoxedCommand, and documentation is available to > help migrate. > > To safely re-enable Score (LilyPond) on Wikimedia wikis, we developed > Shellbox, a way to run shell commands in a remote, isolated container. > This is (hopefully) a stronger level of isolation than we previously had > with firejail, since it's relying on Linux containers and Kubernetes to > do the isolation. At the same time, this helps us in moving towards > running MediaWiki on Kubernetes, as we don't want to include all these > external commands inside the MediaWiki container. For the most part, any > new shelling out to external commands needs to be done via Shellbox. > > A lot of the design and rationale behind Shellbox is captured in the > RfC: <https://phabricator.wikimedia.org/T260330>. > > In Wikimedia production, so far Score, Timeline, SyntaxHighlight and > Wikidata constraint regex checking are all using Shellbox. Details about > that and links to dashboards are available at > <https://wikitech.wikimedia.org/wiki/Shellbox>. The main things that are > left are media-handling code that extracts metadata: DjVu, PdfHandler > and PagedTiffHandler, which is tracked at > <https://phabricator.wikimedia.org/T289228>, and videoscaling > (TimedMediaHandler). > > Some work has to be done in MediaWiki to make code compatible with > Shellbox, specifically switching to "BoxedCommand", which now has its > own documentation page: > <https://www.mediawiki.org/wiki/Manual:BoxedCommand>. BoxedCommand works > transparently whether you have a separate Shellbox service set up or > not. This is the preferred way to write new shellouts going forward, > though Shell::command() isn't officially deprecated yet. So far all > shellouts that are used in Wikimedia production have already been > converted except for TimedMediaHandler. > > Looking forward, I think this also gives us a lot of flexibility in > using more external commands in the future. First, we're less tied to > whatever OS version MediaWiki is running on, as long as it can be > built/shipped in a container, we can use it. And secondly, it's probably > OK if external commands aren't super well behaved (e.g. use too much > memory) since they're no longer sharing the same resources as an > appserver (this shouldn't be interpreted as a free pass for super > inefficient stuff of course). > > I tried to keep this summary short, and am intending to write a longer > blog post that explains some more history in detail. But if you have any > questions or something isn't clear, please ask! > > -- Kunal > _______________________________________________ > Wikitech-l mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >
_______________________________________________ Wikitech-l mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
