Re: [OSM-dev] planet.osm.org is verry slow
Le 08/02/2020 à 10:09, mmd a écrit : On 2020-01-25 16:45, Grant Slater wrote: The 20GB+ planet .osm.bz2 files are automatically redirected to the https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/ mirror But unfortunately due to flaw in my mirror update script the redirect was now working for PBF files. Pull requests welcome: https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb I have manually fixed the PBF file mirroring now (for current files). This is fixed now. I've created a pull request to redirect all planet pbf and full history files to the mirror once they're available. We've also started experimenting bittorrent downloads a few weeks ago. They are available at: https://osm.cquest.org/torrents/ For command line bittorrent downloads you can have a look at lftp and aria2 The latest (today) planet will be available in 3 to 4 hours from now... -- Christian Quest - OpenStreetMap France ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
On 2020-01-25 16:45, Grant Slater wrote: > The 20GB+ planet .osm.bz2 files are automatically redirected to the > https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/ > mirror > But unfortunately due to flaw in my mirror update script the redirect > was now working for PBF files. Pull requests welcome: > https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb > I have manually fixed the PBF file mirroring now (for current files). This is fixed now. I've created a pull request to redirect all planet pbf and full history files to the mirror once they're available. -- ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
OpenMapTiles has a tool to download the planet from all mirrors in parallel. Usually takes a few minutes, and it automatically verifies md5 checksum. The tool will not use the primary planet source by default. Lastly, the tool can also download validated extracts from geofabrik, bbbike, and osmfr. Internally the tool uses aria2c. Easiest is to use it with the docker -- share current dir with docker and save the file there. Anything after the "--" will be passed to aria2c. This is a Linux/Mac command, but should be runnable from Windows too with minor path fixes. docker run --rm -it -v $PWD:/planet openmaptiles/openmaptiles-tools download-osm planet -- -d /planet To run it without the actual download (test run), use --dry-run: docker run --rm -it openmaptiles/openmaptiles-tools download-osm planet --dry-run Use --help for all arguments. See source and documentation here: https://github.com/openmaptiles/openmaptiles-tools#multi-streamed-osm-data-downloader On Sat, Jan 25, 2020 at 5:33 AM Wolfram Schneider wrote: > I wanted to download the latest planet from planet.osm.org and the > download rate is less than 450KByte/s. In the past I got > 30-60Mbyte/s. It takes now more than 36 hours to download the planet > (was 15min). All mirrors are outdated, you have to wait until Sat > morning to see a new planet.osm.pbf file. > > I tried to download the planet from different locations, IP addresses > and clients - it is always slow per connection. It seems that the > slowdown is on purpose by a config change, not an overloaded machine. > > Two weeks ago everything was fine. Does anybody has an idea whats happens > here? > > Thanks, Wolfram > > -- > Planet.osm extracts: https://extract.bbbike.org > BBBike Map Compare: https://mc.bbbike.org > > ___ > dev mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Am 25.01.2020 um 22:19 schrieb marc marc: > Le 25.01.20 à 21:43, Simon Poole a écrit : >> Am 25.01.2020 um 21:12 schrieb marc marc: >>> is there a convenient way to update a pbf to exactly the same diff as >>> osm.org? and/or add the state.txt of the pbf produced on osm.org? >>> this would allow to have the same pbf mirrored on other sites without >>> having to download and/or encode the same thing several times >> The answer is likely no, because I don't believe there is a guarantee >> that the planet dump contains exactly the commits of a certain set of >> diffs, > the planet dump is done from the db itselft and not using by a cascade > of diffs (min->hourly>daily>weekly) ? > or maybe a special diff between planet dump. Yes the dump works via a completely different mechanism. The problem is actually fixable (not expecting a lot of enthusiasm for it though), we could post creating the dump run an update from a specific diff prior to when the dump started on the dump before publishing. Doing this wouldn't guarantee that the binaries of a parallel updated planet file are the same, but they would have the same contents. >> But that doesn't matter in a practical sense > it avoids having different files between mirrors, which is quite > annoying if a bug occurs on one variant and not on another depending > on the "mirror". See above. > >> In any case there is no argument to be made for the ~1'000 downloads >> that start shortly every week after the plant dump has been produced, > I agree. > maybe a rate limit "1st full speed" or "full speed for mirror" AFAIK that is already the case. Simon >> and of which for weird reasons 25% are actually >> downloading the compressed XML file (which is 30% larger). > there are probably some tools that don't support pbf (for example > osmfilter (I don't mind too much, I do an osmconvert before, but I guess > this extra step is not done and/or not available and/or not known by > other people). > >> Simon >> >>> Le 25.01.20 à 18:34, marc tobias a écrit : >>>> Wolfram, who asked about the bandwidth, is already running a mirror >>>> https://download.bbbike.org/osm/planet/ >>>> (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading) >>>> so downloading it every week is likely trying to update his mirror. >>>> >>>> marc tobias >>>> >>>>> Message: 2 >>>>> Date: Sat, 25 Jan 2020 11:46:20 +0100 >>>>> From: Simon Poole >>>>> To: [email protected] >>>>> Subject: Re: [OSM-dev] planet.osm.org is verry slow >>>>> Message-ID: <[email protected]> >>>>> Content-Type: text/plain; charset="utf-8" >>>>> >>>>> Yes, there are now bandwidth restrictions on connections to >>>>> planet.openstreetmap.org because the usage had skyrocketed and outgoing >>>>> bandwidth was completely saturated. >>>>> >>>>> But the obvious question is: why are you even trying to download the >>>>> full planet twice within a fortnight? Particularly given the file can >>>>> easily be updated in situ. >>>>> >>>>> Simon >>>> ___ >>>> dev mailing list >>>> [email protected] >>>> https://lists.openstreetmap.org/listinfo/dev >>>> >>> ___ >>> dev mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/dev > ___ > dev mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/dev signature.asc Description: OpenPGP digital signature ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Le 25.01.20 à 21:43, Simon Poole a écrit : > > Am 25.01.2020 um 21:12 schrieb marc marc: >> is there a convenient way to update a pbf to exactly the same diff as >> osm.org? and/or add the state.txt of the pbf produced on osm.org? >> this would allow to have the same pbf mirrored on other sites without >> having to download and/or encode the same thing several times > > The answer is likely no, because I don't believe there is a guarantee > that the planet dump contains exactly the commits of a certain set of > diffs, the planet dump is done from the db itselft and not using by a cascade of diffs (min->hourly>daily>weekly) ? or maybe a special diff between planet dump. > But that doesn't matter in a practical sense it avoids having different files between mirrors, which is quite annoying if a bug occurs on one variant and not on another depending on the "mirror". > In any case there is no argument to be made for the ~1'000 downloads > that start shortly every week after the plant dump has been produced, I agree. maybe a rate limit "1st full speed" or "full speed for mirror" > and of which for weird reasons 25% are actually > downloading the compressed XML file (which is 30% larger). there are probably some tools that don't support pbf (for example osmfilter (I don't mind too much, I do an osmconvert before, but I guess this extra step is not done and/or not available and/or not known by other people). > > Simon > >> >> Le 25.01.20 à 18:34, marc tobias a écrit : >>> Wolfram, who asked about the bandwidth, is already running a mirror >>> https://download.bbbike.org/osm/planet/ >>> (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading) >>> so downloading it every week is likely trying to update his mirror. >>> >>> marc tobias >>> >>>> Message: 2 >>>> Date: Sat, 25 Jan 2020 11:46:20 +0100 >>>> From: Simon Poole >>>> To: [email protected] >>>> Subject: Re: [OSM-dev] planet.osm.org is verry slow >>>> Message-ID: <[email protected]> >>>> Content-Type: text/plain; charset="utf-8" >>>> >>>> Yes, there are now bandwidth restrictions on connections to >>>> planet.openstreetmap.org because the usage had skyrocketed and outgoing >>>> bandwidth was completely saturated. >>>> >>>> But the obvious question is: why are you even trying to download the >>>> full planet twice within a fortnight? Particularly given the file can >>>> easily be updated in situ. >>>> >>>> Simon >>> ___ >>> dev mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/dev >>> >> ___ >> dev mailing list >> [email protected] >> https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Am 25.01.2020 um 21:12 schrieb marc marc: > is there a convenient way to update a pbf to exactly the same diff as > osm.org? and/or add the state.txt of the pbf produced on osm.org? > this would allow to have the same pbf mirrored on other sites without > having to download and/or encode the same thing several times The answer is likely no, because I don't believe there is a guarantee that the planet dump contains exactly the commits of a certain set of diffs, so I suspect you are asking for something that the planet dump itself doesn't actually provide (Matt Amos would be able to give more information on that, see https://github.com/zerebubuth/planet-dump-ng). But that doesn't matter in a practical sense, as you will always roll back a bit the 1st time you start applying diffs and from then on your planet will exactly contain a diff or not. There is perhaps an argument to be made that actual mirrors should make a binary copy. In any case there is no argument to be made for the ~1'000 downloads that start shortly every week after the plant dump has been produced, and of which for weird reasons 25% are actually downloading the compressed XML file (which is 30% larger). Simon > > Le 25.01.20 à 18:34, marc tobias a écrit : >> Wolfram, who asked about the bandwidth, is already running a mirror >> https://download.bbbike.org/osm/planet/ >> (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading) >> so downloading it every week is likely trying to update his mirror. >> >> marc tobias >> >>> Message: 2 >>> Date: Sat, 25 Jan 2020 11:46:20 +0100 >>> From: Simon Poole >>> To: [email protected] >>> Subject: Re: [OSM-dev] planet.osm.org is verry slow >>> Message-ID: <[email protected]> >>> Content-Type: text/plain; charset="utf-8" >>> >>> Yes, there are now bandwidth restrictions on connections to >>> planet.openstreetmap.org because the usage had skyrocketed and outgoing >>> bandwidth was completely saturated. >>> >>> But the obvious question is: why are you even trying to download the >>> full planet twice within a fortnight? Particularly given the file can >>> easily be updated in situ. >>> >>> Simon >> ___ >> dev mailing list >> [email protected] >> https://lists.openstreetmap.org/listinfo/dev >> > ___ > dev mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/dev signature.asc Description: OpenPGP digital signature ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
is there a convenient way to update a pbf to exactly the same diff as osm.org? and/or add the state.txt of the pbf produced on osm.org? this would allow to have the same pbf mirrored on other sites without having to download and/or encode the same thing several times Le 25.01.20 à 18:34, marc tobias a écrit : > Wolfram, who asked about the bandwidth, is already running a mirror > https://download.bbbike.org/osm/planet/ > (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading) > so downloading it every week is likely trying to update his mirror. > > marc tobias > >> Message: 2 >> Date: Sat, 25 Jan 2020 11:46:20 +0100 >> From: Simon Poole >> To: [email protected] >> Subject: Re: [OSM-dev] planet.osm.org is verry slow >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset="utf-8" >> >> Yes, there are now bandwidth restrictions on connections to >> planet.openstreetmap.org because the usage had skyrocketed and outgoing >> bandwidth was completely saturated. >> >> But the obvious question is: why are you even trying to download the >> full planet twice within a fortnight? Particularly given the file can >> easily be updated in situ. >> >> Simon > > ___ > dev mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Wolfram, who asked about the bandwidth, is already running a mirror https://download.bbbike.org/osm/planet/ (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading) so downloading it every week is likely trying to update his mirror. marc tobias > Message: 2 > Date: Sat, 25 Jan 2020 11:46:20 +0100 > From: Simon Poole > To: [email protected] > Subject: Re: [OSM-dev] planet.osm.org is verry slow > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8" > > Yes, there are now bandwidth restrictions on connections to > planet.openstreetmap.org because the usage had skyrocketed and outgoing > bandwidth was completely saturated. > > But the obvious question is: why are you even trying to download the > full planet twice within a fortnight? Particularly given the file can > easily be updated in situ. > > Simon ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
On 25/01/2020 15:45, Grant Slater wrote: The minutely, hourly and daily diff files will only take a second or 2 extra to download with the current late limits. The minutely files won't normally be affected at all as the limit only kicks in for files over 512Kb. Tom -- Tom Hughes ([email protected]) http://compton.nu/ ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Hi Wolfram and others, On Sat, 25 Jan 2020 at 10:32, Wolfram Schneider wrote: > > I wanted to download the latest planet from planet.osm.org and the > download rate is less than 450KByte/s. In the past I got > 30-60Mbyte/s. It takes now more than 36 hours to download the planet > (was 15min). All mirrors are outdated, you have to wait until Sat > morning to see a new planet.osm.pbf file. > I am part of the OpenStreetMap Operations team and Operations Working Group. I am the person who temporarily instigated the download rate limit. https://github.com/openstreetmap/chef/commit/1647548590878bfd153b8b8c74e73eda0b8a2cd3 Some background... www.openstreetmap.org + API, planet and many other services primarily run in our Amsterdam data center (Equinix AM6). See servers here: https://hardware.openstreetmap.org/#equinix-am6 Our uplink from these servers to the internet is a 1Gbps link to Cogent. Bandwidth usage: https://munin.osm.org/openstreetmap.org/switch1.openstreetmap.org/if_bytes/if_49.html Due to recent increases in outbound bandwidth usage (partially due to switch to BBR congestion control https://github.com/openstreetmap/chef/commit/67c05c6c4b484a6cdac9473b4e6951e5c76c42a3), we started to exceed 1Gbps... Once we exceed the limit, packets get dropped. We initially did not understand what was causing the packets to be dropped. We suspected a flaw with switches and/or GBIC modules and/or fibre link. The dropped packet caused DNS failures and reliability issues with the API, which were reported in many channels: email, Slack osm-us, irc etc. The planet downloads are the main reason the upstream link is being swamped. 60Mbyte/s alone is nearly 50% of our available upstream capacity. Normally around 30 to 60 simultaneous requests. I've started to prep for upgrading our upstream link to 10Gbps. But we've had ongoings with Cogent peering and would prefer to switch to using a new provider upstream. https://twitter.com/OSM_Tech/status/1220276733359919104 The 20GB+ planet .osm.bz2 files are automatically redirected to the https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/ mirror But unfortunately due to flaw in my mirror update script the redirect was now working for PBF files. Pull requests welcome: https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb I have manually fixed the PBF file mirroring now (for current files). Once we have fixed the upstream link issue, I will remove the download rate limits. For the moment, please use a mirror for full speed downloads. The minutely, hourly and daily diff files will only take a second or 2 extra to download with the current late limits. Kind regards, Grant ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
As Simon mentioned, the planet server itself is bandwidth limited now. Can you use a planet mirror? It should redirect you to a mirror when requesting a download. If it isn't, maybe there's a bug to resolve. On Sat, Jan 25, 2020, 04:33 Wolfram Schneider wrote: > I wanted to download the latest planet from planet.osm.org and the > download rate is less than 450KByte/s. In the past I got > 30-60Mbyte/s. It takes now more than 36 hours to download the planet > (was 15min). All mirrors are outdated, you have to wait until Sat > morning to see a new planet.osm.pbf file. > > I tried to download the planet from different locations, IP addresses > and clients - it is always slow per connection. It seems that the > slowdown is on purpose by a config change, not an overloaded machine. > > Two weeks ago everything was fine. Does anybody has an idea whats happens > here? > ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] planet.osm.org is verry slow
Yes, there are now bandwidth restrictions on connections to planet.openstreetmap.org because the usage had skyrocketed and outgoing bandwidth was completely saturated. But the obvious question is: why are you even trying to download the full planet twice within a fortnight? Particularly given the file can easily be updated in situ. Simon Am 25.01.2020 um 11:29 schrieb Wolfram Schneider: > I wanted to download the latest planet from planet.osm.org and the > download rate is less than 450KByte/s. In the past I got > 30-60Mbyte/s. It takes now more than 36 hours to download the planet > (was 15min). All mirrors are outdated, you have to wait until Sat > morning to see a new planet.osm.pbf file. > > I tried to download the planet from different locations, IP addresses > and clients - it is always slow per connection. It seems that the > slowdown is on purpose by a config change, not an overloaded machine. > > Two weeks ago everything was fine. Does anybody has an idea whats happens > here? > > Thanks, Wolfram > > -- > Planet.osm extracts: https://extract.bbbike.org > BBBike Map Compare: https://mc.bbbike.org > > ___ > dev mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/dev signature.asc Description: OpenPGP digital signature ___ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
