I realise now I did in fact send my last email to the list, rather than
to Patrick directly....no harm done! The info might be useful to others
anyway.
Thanks Bjorn, that's very helpful. Our extract took just over 7 hours
yesterday, which isn't as long as I thought it would take, so we'll
probably just schedule it to run every weekend or so and move the files
to the correct location when finished.
Kind regards,
Kieran Caplice
On 03/03/16 09:23, Björn Semm wrote:
Hi Kieran,
we run an OSRM update (planet) once a week on a central instance and copy the
generated files to diffrent environments.
osrm@box:~$ ./osrm-update-planet-files.sh
Checking for md5sum [OK]
Checking for osrm-extract [OK]
Checking for osrm-prepare [OK]
Checking for tar [OK]
Checking for wget [OK]
Downloading planet-latest.osm.pbf.md5 ... [OK]
Downloading http://planet.osm.org/pbf/planet-latest.osm.pbf ... [OK]
Verifying md5 checksum of planet-latest.osm.pbf ... [OK]
Starting osrm-extract at Wed Mar 2 11:57:41 CET 2016...
Finished osrm-extract at Thu Mar 3 00:21:34 CET 2016!
Starting osrm-prepare at Thu Mar 3 00:21:34 CET 2016...
Finished osrm-prepare at Thu Mar 3 09:21:23 CET 2016!
Removing old extracts from /data/current ... empty [OK]
Copying new generated files to /data/current ... [OK]
Renaming files in /data/current with Prefix 201609 ... [OK]
Creating md5 checksum over all 201609_planet-latest* ... [OK]
Compressing 201609_planet-latest* to 201609_planet-latest.tar.gz ... [OK]
Determining if test or prod env is the target ... TEST [OK]
Copying new generated files to /mnt/osrm-extract (TEST) ... [OK]
Cleaning up /mnt/osrm-extract ... [OK]
Cleanup /data/planet-latest.osm.pbf ... [OK]
On a VM with 96GB RAM, 4 Cores and a RAID5 (HDD) it took about 12,5 hours to
extract and 9 hours to prepare.
SWAP is 100GB, stxxl=disk=/data/stxxl,250000,syscall
We currently use Version 4.9.0 of osrm-backend.
BR
Björn
________________________________________
Von: Kieran Caplice <[email protected]>
Gesendet: Mittwoch, 2. März 2016 18:23
An: [email protected]
Betreff: Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Patrick,
That makes sense then. It's obvious the process is just going to take
upwards of 8-10 hours for us in that case.
Thanks for the help.
Kind regards,
Kieran Caplice
On 02/03/16 17:01, Patrick Niklaus wrote:
Hey Kieran,
there have been a lot of structural changes (e.g. moving code from
osrm-prepare into osrm-extract) that probably invalidate that numbers.
Also we support 64bit OSM ids now, which sadly uses a lot more disk
space. I think stxxl need like 200GB. I think on our setup we have a
turn-around of 6 hours for the planet dataset on an SSD setup (car
profile, any other profile needs significantly longer). You should
probably think about updating your hard drives as this is IO bound. At
your current read/write speed it will already take more than an hour
to just write 200GB of data once. We scan it at least twice just for
pre-processing.
Cheers,
Patrick
On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
<[email protected]> wrote:
Hello,
I'm currently extracting the planet PBF (~31 GB), and it's been running for
hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
to do so from a PBF formatted planet", which is making me wonder why it's
taking so long on our server. Below are some example output messages:
[info] Parsing finished after 3584.35 seconds
[extractor] Erasing duplicate nodes ... ok, after 319.091s
[extractor] Sorting all nodes ... ok, after 3632.87s
[extractor] Building node id map ... ok, after 2025.29s
[extractor] Confirming/Writing used nodes ... ok, after 1096.24s
[extractor] Sorting edges by start ... ok, after 2000.08s
Some stxxl errors were outputted as I set the disk size to 100GB thinking it
was enough - but I didn't think it would cause such slowdowns as this,
considering extracting the Europe PBF takes hours also without the stxxl
errors.
Server specs:
Ubuntu 14.04
Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT)
64 GB RAM @ 2133 MHz
2 TB Western Digital Enterprise 7200 RPM hard drive
At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
Anyone have any ideas as to what might be going on? Or is it normal to take
this long without an SSD?
Thanks in advance.
Kind regards,
Kieran Caplice
_______________________________________________
OSRM-talk mailing list
[email protected]
https://lists.openstreetmap.org/listinfo/osrm-talk
_______________________________________________
OSRM-talk mailing list
[email protected]
https://lists.openstreetmap.org/listinfo/osrm-talk
_______________________________________________
OSRM-talk mailing list
[email protected]
https://lists.openstreetmap.org/listinfo/osrm-talk
_______________________________________________
OSRM-talk mailing list
[email protected]
https://lists.openstreetmap.org/listinfo/osrm-talk
