Not sure what you mean exactly with "coordinate". You can write a tool which can do that because both expose everything via API.
On our side, with farm.openzim.org, we have already a few third party orgs which use it for their own stuff. Emmanuel On 24.06.20 16:09, Samuel Klein wrote: > Wow, this is fabulous. If a new zimfarm starts up, can it coordinate > with existing ones? > > On Tue, Jun 23, 2020 at 3:23 AM Emmanuel Engelhart <[email protected] > <mailto:[email protected]>> wrote: > > Hi > > There is a topic I wanted to talk about here for a long time and for > which I never have achieved to take the time to write something. A few > recent events have been a healthy remember that I should present one our > most recent and most useful tool: Zimfarm. > > The Zimfarm is the online tool which is in charge of building and > publishing all our ZIM files. After years of creating ZIM files by > launching scrapers more or less manually, we had to automatise the > process to just be able to scale the operations, ie. publishing more and > more often ZIM files. > > The effort started 3 years ago with the support of the WMF but we use it > only since Spring 2019 in production. The tool is now perfectly running > and we fully rely on it now. If we can publish an update of all our > wikis one time a month, this is thanks to this piece of software too. > > The Zimfarm is a half-decentralized solution which has a central node > (called "dispatcher") in charge of orchestrating the work to do and > multiple decentralized nodes (called "workers") which run the scraping > tasks. > > The dispatcher provides an API to manage the ZIM recipes and tasks, have > a look to https://api.farm.openzim.org/. We have setup a Web frontend on > this API to allow easy mgmt through a Web browser. For a better > transparency, even anonymous users can have a look and monitor what is > going on. Look at https://farm.openzim.org/. > > One important point is that, like all the rest of our infrastructure, > the whole system is Dockerized. Which means, this is really easy to > install a Zimfarm worker and we invite anybody having a spare server to > help us to provide offline snapshots of the best of the Web. The > procedure is documented and a few volunteers have already joined in. > Look at https://farm.openzim.org/about for more details. > > The development is fully transparent at > https://github.com/openzim/zimfarm. We have a few things which are on > the roadmap which would welcome volunteer Python developers. Look at the > good first issues and make your first PR! > > https://github.com/openzim/zimfarm/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 > > Regards > Emmanuel > > -- > Kiwix - Wikipedia Offline & more > * Web: https://kiwix.org/ > * Twitter: https://twitter.com/KiwixOffline > * Wiki: https://wiki.kiwix.org/ > > _______________________________________________ > Offline-l mailing list > [email protected] <mailto:[email protected]> > https://lists.wikimedia.org/mailman/listinfo/offline-l > > > > -- > Samuel Klein @metasj w:user:sj +1 617 529 4266 > > _______________________________________________ > Offline-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/offline-l > -- Kiwix - Wikipedia Offline & more * Web: https://kiwix.org/ * Twitter: https://twitter.com/KiwixOffline * Wiki: https://wiki.kiwix.org/
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Offline-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/offline-l
