ArielGlenn added a comment.
In T222349#5753559 <https://phabricator.wikimedia.org/T222349#5753559>, @Gehel wrote: > In T222349#5751029 <https://phabricator.wikimedia.org/T222349#5751029>, @ArielGlenn wrote: > >> How fast a download do folks want? > > As fast as possible! using an external mirror, we get a download speed of ~60M/s, which means that a Wikidata dump can be downloaded in ~15 minutes. From our own dumps servers, at 5M/s this takes about 3 hours. So ideally the same speed would be great. We would already switch to using our own dump servers even with 2x slower than external. > >> Can we schedule rsyncs for the specifiic use cases with a higher bandwidth cap? > > Would rsync put less load on the dump servers? From my point of view, we don't care much about rsync vs curl. The thing about rsync is that we could set up a job to update just the dumps required at the interval wanted, with a single connection open. I'd feel a lot better about high bandwidth limits for something like that than generally opening up web service. TASK DETAIL https://phabricator.wikimedia.org/T222349 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArielGlenn Cc: EBernhardson, Bstorm, ArielGlenn, Gehel, Aklapper, Phamhi, darthmon_wmde, Legado_Shulgin, Nandana, Killertrap, sietec, Jony, Zylc, Giuliamocci, Davinaclare77, Qtn1293, 1978Gage2001, Techguru.pc, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Allthingsgo, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, srodlund, Wong128hk, mys_721tx, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, Jitrixis, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Rxy, Jay8g, bd808, Krenair, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
