On 02/12/2015 02:43 AM, Stephen John Smoogen wrote: > - Mirror versioning: > - run UMDL, detect changes, increase master mirror's version by 1 > - run the crawler, check for the changes, align that mirror's > version with the > master mirror one > - be able to see the difference between two versions > - be able to crawl a mirror only for the difference between the > version it is > at and the version the master mirror is at > note: we might still want to run a full crawl once in a while (daily? > bi-daily?) > > > Those last couple of steps would be very useful in cutting down load > from mirrors running rsync against enchillada but only needing a couple > of packages in updates since the last time they did it.
In that general vein, an interesting project Robert Collins introduced me to last year is his lmirror work: https://pypi.python.org/pypi/lmirror It's designed to take advantage of various HTTP/2 features and the knowledge that it's specifically being used to create read-only mirrors to make the network operations more efficient than is feasible with the more general purpose rsync. Design details: https://bazaar.launchpad.net/~lmirror/lmirror/master/view/head:/doc/DESIGN.txt Regards, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
