A similar discussion has happened recently when BitTorrent downloads were announced as an experimental feature (wiki discussion <https://wiki.openstreetmap.org/wiki/Talk:Planet.osm#Torrent>, twitter <https://twitter.com/OSM_Tech/status/1222507257478422535>, reddit <https://www.reddit.com/r/openstreetmap/comments/evmqbb/osm_operations_team_we_will_soon_be_adding/>, yc <https://news.ycombinator.com/item?id=7171847>) and the reception seems to be pretty good. My understanding is that there's a use case for projects that don't need to have a permanently updated planet but want to do a full update in a low pace frequency (say twice per year or so). Helping those projects to achieve that part using a parallel download through several mirrors (or BitTorrent) makes sense to me. The increase of bandwidth, and eventually the need for more mirrors/peers could be considered a correlation with the increase of adoption of OSM data, right?
Also, this tool is also meant to help to automate the download of data from other extract sources (geofabrik, openstreetmap fr, bbbike) for smaller areas, which is also useful in itself for projects aiming at a smaller geographical context. On Tue, 4 Feb 2020 at 01:42, Yuri Astrakhan <yuriastrak...@gmail.com> wrote: > Andy, I agree that being frugal with bandwidth is important. Yet, there > is a significant operations cost involved here, that I suspect very few > will actually be willing to spend, unless it is made trivial -- the cost of > setting up an independent planet file update pipeline - i.e. a docker image > that could be pointed to a planet file, and has an easy way to suspend and > resume update when the file needs to be static during the loading process. > > I think having an optimized download tool that can both download+validate > planet/areas, and could provide other services like diff updating would > solve both goals. If the current process is manually pick a mirror, > manually validate, and re-download if failed, vs use a dedicated tool that > will optimize the download and crash recovery, all the better. Especially > if it offers us a way to add more functionality later, like patch updating. > > Also, lets not fall into the premature optimization trap, as that only > achieves local rather than global maximums. As a theoretical example - > what is better - wider OSM adaption with higher number of planet downloads > (i.e. some wasted bandwidth), or lower adaption and more optimized download > process? I would think OSM project would be better off from wider adaption > at a cost of a 10-50% extra bandwidth, right? If the startup cost (human > hours) is lower, more people would participate. My numbers could be > totally off, but keeping the eyes on bigger picture is very important when > deciding such matters. Convenience/low manual labor cost is a big > contributor to wider adaption. > > P.S. Am I correct to assume that every data consumer would need to > download each diff file twice -- once for planet file update, and once to > update the PostgreSQL database? > > > On Sun, Feb 2, 2020 at 4:17 PM Andy Townsend <ajt1...@gmail.com> wrote: > >> ... that's why I said "... and there are ways of keeping a *.pbf* up to >> date" in my message. - the idea is to avoid large downloads wherever >> possible (emphasis new in this message; won't make it to plain text >> archive). >> > Andy, > > >> For more info see: >> >> https://docs.osmcode.org/pyosmium/latest/tools_uptodate.html >> >> or if that's somehow not an option: >> >> >> https://wiki.openstreetmap.org/wiki/User:EdLoach#Osmosis_to_keep_a_local_copy_of_a_.osm_file_updated >> >> (from a few years back) >> >> * Automation / easy adaptation. Providing an out-of-the box way to set >> up your own server is much easier if you have a tool that automatically >> downloads and validates the planet file or a portion of it, >> >> Sure - an automated process is much easier to follow than a manual >> do-it-yourself one, but the fact that a process is automated doesn't mean >> that it has to be wasteful. Ultimately, someone's paying for the bandwidth >> that downloads from each of the mirrors at >> https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors use, >> so it seems only fair to not be profligate with it. >> >> Best Regards, >> >> Andy >> >> >> >> _______________________________________________ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > -- Jorge Sanz http://twitter.com/xurxosanz http://jorgesanz.net
_______________________________________________ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
