Re: [Foundation-l] [Xmldatadumps-l] Wikipedia dumps downloader
On Tue, Jun 28, 2011 at 10:49 PM, Peter Gervai grin...@gmail.com wrote: On Tue, Jun 28, 2011 at 10:12, Milos Rancic mill...@gmail.com wrote: On 06/27/2011 01:07 PM, emijrp wrote: If you create a way to sync images partially (let's say, the base letters inside of the directory structure), a couple of dozens of Wikimedians could create a couple of dozens of backups. I could host on my personal computer ~500GB, which approximates one letter. Some chapters have servers with storage space, too. Being able to keep a backup of all media under Category:Australia on an Australian server would be beneficial, as there are occasional calls for this in order to keep the Australian content 'safe'. If this was possible to backup only the works related to Australia, we could also ask the national library ([[PANDORA]] project) if they consider it within their scope to back it up. -- John Vandenberg ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] [Xmldatadumps-l] Wikipedia dumps downloader
On 06/27/2011 01:07 PM, emijrp wrote: Yes, a distributed project would be probably the best solution, but it is not easy to develop, unless you use a library like bittorrent, or similar and you have many peers. Althought most of the people don't seed the files long time, so sometimes is better to depend on a few committed persons than a big but ephemeral crowd. If you create a way to sync images partially (let's say, the base letters inside of the directory structure), a couple of dozens of Wikimedians could create a couple of dozens of backups. I could host on my personal computer ~500GB, which approximates one letter. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] [Xmldatadumps-l] Wikipedia dumps downloader
On Tue, Jun 28, 2011 at 10:12, Milos Rancic mill...@gmail.com wrote: On 06/27/2011 01:07 PM, emijrp wrote: If you create a way to sync images partially (let's say, the base letters inside of the directory structure), a couple of dozens of Wikimedians could create a couple of dozens of backups. I could host on my personal computer ~500GB, which approximates one letter. Some chapters have servers with storage space, too. g ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] [Xmldatadumps-l] Wikipedia dumps downloader
Hi Richard; Yes, a distributed project would be probably the best solution, but it is not easy to develop, unless you use a library like bittorrent, or similar and you have many peers. Althought most of the people don't seed the files long time, so sometimes is better to depend on a few committed persons than a big but ephemeral crowd. Regards, emijrp 2011/6/26 Richard Farmbrough rich...@farmbrough.co.uk ** It would be useful to have an archive of archives. I have to delete my old data dumps as time passes, for space reasons, however a team could, between them, maintain multiple copies of every data dump. This would make a nice distributed project. On 26/06/2011 13:53, emijrp wrote: Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps (and her sister projects) from dumps.wikimedia.org. It sorts in folders and checks md5sum. It only works on Linux (it uses wget). You will need about 100GB to download all the 7z files. Save our memory. Regards, emijrp [1] http://en.wikipedia.org/wiki/Destruction_of_libraries [2] http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py ___ Xmldatadumps-l mailing listXmldatadumps-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
