| aborrero added a subscriber: bd808. aborrero added a comment. |
In T195121#4424983, @abian wrote:In T195121#4224564, @aborrero wrote:
- Is it possible to do the processing by type (your first 3 points) instead of all at the same time? So you could reuse small chunks of storage instead of having to allocate a big one
We'll definitely do the processing by type to reuse small chunks of storage, yes.
ACK
- How do they offer the datasets? Do they have an API you can query for more data? Do they offer a single big file for downloading with all the data? Depending on this, our approach could be very different. We could build a "pipeline" instead of working with large batches
The data are spread through several downloadable files. I think I'll be able to structure these data somehow.
Do you have an estimation on how big is each downloadable file? You mentioned a series of raster topographic maps at a scale of 1:25, formats TIFF and ECW (~785 GB),, but I guess this is composed of several smaller files.
Let's assume it's 10%, i.e. 79GB. We could allocate a ~200GB VM for you to download from there and do the processing, upload to commons, clean and continue with the next block.
- Do you have an estimation on how many time all the operations would take? i.e, how much time should we provide the storage facilities? 1 week, 1 month, 6 month, 1 year...
No, for now we can't guess how many time the operations would take since we still don't know what's the best way to transfer the files from a hard drive to Commons. The points are:
- We have to buy a hard drive so that the IGN can store all the files in it. Do we need a SSD to be able to transfer/upload the files within a reasonable time but at the expense of having a limited capacity? Would a HDD be enough? Will this difference in time be actually noticeable?
- How do we transfer the files in the SSD/HDD to the Cloud VPS? Can we physically send the drive to the location of the server? Or should we transfer the files via the Internet? Considering the bandwith and the size of the files, is this last option possible? @aborrero, maybe you can help us with some of these points?
Networking (specially international) will surely be a bottleneck. Depending on the file size (and protocol), data transfer might be very painfull. We may use rsync or some bittorrent protocol to ensure that even with long transfer times we are able to actually move the data from one point to the other.
I don't think moving around a physical drive would help us in this case (and I ignore which policy we have to plug a random drive in one of our servers).
So, this is my proposal, assuming ~79GB data files:
- let's allocate a VM with a 200GB extra disk (to be discussed with @bd808 and the rest of the WMCS team)
- let's give you access to this VM
- start testing our pipeline
- download of a single ~79GB data file from IGN, using something like rsync, torrent, etc
- processing
- uploading to commons
- clean VM to leave room for next block
Cc: bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
