| aborrero added a comment. |
Hey, this is great news, since IGN is indeed a powerful source of data regarding geography, topology and mountains (which I love).
From the Wikimedia Cloud Services point of view, one thing we could offer is a Cloud VPS temporal project, where you can create one or two virtual machines to do all this work. We can delete the project when you are done.
If you plan to hold all the data at the same time, it seems you need about 3TB just to store what IGN offers. Then I guess you need at least the same for processing the data and the intermediate step before uploading to Wikimedia Commons.
This is about 6TB storage, which is not trivial to allocate I think.
Some questions:
- Do you have an estimation on how many time all the operations would take? i.e, how much time should we provide the storage facilities? 1 week, 1 month, 6 month, 1 year...
- Is it possible to do the processing by type (your first 3 points) instead of all at the same time? So you could reuse small chunks of storage instead of having to allocate a big one
- How do they offer the datasets? Do they have an API you can query for more data? Do they offer a single big file for downloading with all the data? Depending on this, our approach could be very different. We could build a "pipeline" instead of working with large batches
TASK DETAIL
EMAIL PREFERENCES
To: aborrero
Cc: aborrero, Aklapper, SandraF_WMF, Platonides, Rodelar, abian, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, bd808, Krenair, chasemp
Cc: aborrero, Aklapper, SandraF_WMF, Platonides, Rodelar, abian, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, bd808, Krenair, chasemp
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
