Re: [OSRM-talk] osrm-extract taking hours to complete
I realise now I did in fact send my last email to the list, rather than to Patrick directlyno harm done! The info might be useful to others anyway. Thanks Bjorn, that's very helpful. Our extract took just over 7 hours yesterday, which isn't as long as I thought it would take, so we'll probably just schedule it to run every weekend or so and move the files to the correct location when finished. Kind regards, Kieran Caplice On 03/03/16 09:23, Björn Semm wrote: Hi Kieran, we run an OSRM update (planet) once a week on a central instance and copy the generated files to diffrent environments. osrm@box:~$ ./osrm-update-planet-files.sh Checking for md5sum [OK] Checking for osrm-extract [OK] Checking for osrm-prepare [OK] Checking for tar [OK] Checking for wget [OK] Downloading planet-latest.osm.pbf.md5 ... [OK] Downloading http://planet.osm.org/pbf/planet-latest.osm.pbf ... [OK] Verifying md5 checksum of planet-latest.osm.pbf ... [OK] Starting osrm-extract at Wed Mar 2 11:57:41 CET 2016... Finished osrm-extract at Thu Mar 3 00:21:34 CET 2016! Starting osrm-prepare at Thu Mar 3 00:21:34 CET 2016... Finished osrm-prepare at Thu Mar 3 09:21:23 CET 2016! Removing old extracts from /data/current ... empty [OK] Copying new generated files to /data/current ... [OK] Renaming files in /data/current with Prefix 201609 ... [OK] Creating md5 checksum over all 201609_planet-latest* ... [OK] Compressing 201609_planet-latest* to 201609_planet-latest.tar.gz ... [OK] Determining if test or prod env is the target ... TEST [OK] Copying new generated files to /mnt/osrm-extract (TEST) ... [OK] Cleaning up /mnt/osrm-extract ... [OK] Cleanup /data/planet-latest.osm.pbf ... [OK] On a VM with 96GB RAM, 4 Cores and a RAID5 (HDD) it took about 12,5 hours to extract and 9 hours to prepare. SWAP is 100GB, stxxl=disk=/data/stxxl,25,syscall We currently use Version 4.9.0 of osrm-backend. BR Björn Von: Kieran Caplice <kieran.capl...@temetra.com> Gesendet: Mittwoch, 2. März 2016 18:23 An: osrm-talk@openstreetmap.org Betreff: Re: [OSRM-talk] osrm-extract taking hours to complete Hi Patrick, That makes sense then. It's obvious the process is just going to take upwards of 8-10 hours for us in that case. Thanks for the help. Kind regards, Kieran Caplice On 02/03/16 17:01, Patrick Niklaus wrote: Hey Kieran, there have been a lot of structural changes (e.g. moving code from osrm-prepare into osrm-extract) that probably invalidate that numbers. Also we support 64bit OSM ids now, which sadly uses a lot more disk space. I think stxxl need like 200GB. I think on our setup we have a turn-around of 6 hours for the planet dataset on an SSD setup (car profile, any other profile needs significantly longer). You should probably think about updating your hard drives as this is IO bound. At your current read/write speed it will already take more than an hour to just write 200GB of data once. We scan it at least twice just for pre-processing. Cheers, Patrick On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice <kieran.capl...@temetra.com> wrote: Hello, I'm currently extracting the planet PBF (~31 GB), and it's been running for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes to do so from a PBF formatted planet", which is making me wonder why it's taking so long on our server. Below are some example output messages: [info] Parsing finished after 3584.35 seconds [extractor] Erasing duplicate nodes ... ok, after 319.091s [extractor] Sorting all nodes ... ok, after 3632.87s [extractor] Building node id map ... ok, after 2025.29s [extractor] Confirming/Writing used nodes ... ok, after 1096.24s [extractor] Sorting edges by start... ok, after 2000.08s Some stxxl errors were outputted as I set the disk size to 100GB thinking it was enough - but I didn't think it would cause such slowdowns as this, considering extracting the Europe PBF takes hours also without the stxxl errors. Server specs: Ubuntu 14.04 Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) 64 GB RAM @ 2133 MHz 2 TB Western Digital Enterprise 7200 RPM hard drive At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). Anyone have any ideas as to what might be going on? Or is it normal to take this long without an SSD? Thanks in advance. Kind regards, Kieran Caplice ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list O
Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Kieran, we run an OSRM update (planet) once a week on a central instance and copy the generated files to diffrent environments. osrm@box:~$ ./osrm-update-planet-files.sh Checking for md5sum [OK] Checking for osrm-extract [OK] Checking for osrm-prepare [OK] Checking for tar [OK] Checking for wget [OK] Downloading planet-latest.osm.pbf.md5 ... [OK] Downloading http://planet.osm.org/pbf/planet-latest.osm.pbf ... [OK] Verifying md5 checksum of planet-latest.osm.pbf ... [OK] Starting osrm-extract at Wed Mar 2 11:57:41 CET 2016... Finished osrm-extract at Thu Mar 3 00:21:34 CET 2016! Starting osrm-prepare at Thu Mar 3 00:21:34 CET 2016... Finished osrm-prepare at Thu Mar 3 09:21:23 CET 2016! Removing old extracts from /data/current ... empty [OK] Copying new generated files to /data/current ... [OK] Renaming files in /data/current with Prefix 201609 ... [OK] Creating md5 checksum over all 201609_planet-latest* ... [OK] Compressing 201609_planet-latest* to 201609_planet-latest.tar.gz ... [OK] Determining if test or prod env is the target ... TEST [OK] Copying new generated files to /mnt/osrm-extract (TEST) ... [OK] Cleaning up /mnt/osrm-extract ... [OK] Cleanup /data/planet-latest.osm.pbf ... [OK] On a VM with 96GB RAM, 4 Cores and a RAID5 (HDD) it took about 12,5 hours to extract and 9 hours to prepare. SWAP is 100GB, stxxl=disk=/data/stxxl,25,syscall We currently use Version 4.9.0 of osrm-backend. BR Björn Von: Kieran Caplice <kieran.capl...@temetra.com> Gesendet: Mittwoch, 2. März 2016 18:23 An: osrm-talk@openstreetmap.org Betreff: Re: [OSRM-talk] osrm-extract taking hours to complete Hi Patrick, That makes sense then. It's obvious the process is just going to take upwards of 8-10 hours for us in that case. Thanks for the help. Kind regards, Kieran Caplice On 02/03/16 17:01, Patrick Niklaus wrote: > Hey Kieran, > > > there have been a lot of structural changes (e.g. moving code from > osrm-prepare into osrm-extract) that probably invalidate that numbers. > Also we support 64bit OSM ids now, which sadly uses a lot more disk > space. I think stxxl need like 200GB. I think on our setup we have a > turn-around of 6 hours for the planet dataset on an SSD setup (car > profile, any other profile needs significantly longer). You should > probably think about updating your hard drives as this is IO bound. At > your current read/write speed it will already take more than an hour > to just write 200GB of data once. We scan it at least twice just for > pre-processing. > > Cheers, > Patrick > > > On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice > <kieran.capl...@temetra.com> wrote: >> Hello, >> >> I'm currently extracting the planet PBF (~31 GB), and it's been running for >> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with >> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes >> to do so from a PBF formatted planet", which is making me wonder why it's >> taking so long on our server. Below are some example output messages: >> >> [info] Parsing finished after 3584.35 seconds >> [extractor] Erasing duplicate nodes ... ok, after 319.091s >> [extractor] Sorting all nodes ... ok, after 3632.87s >> [extractor] Building node id map ... ok, after 2025.29s >> [extractor] Confirming/Writing used nodes ... ok, after 1096.24s >> [extractor] Sorting edges by start... ok, after 2000.08s >> >> Some stxxl errors were outputted as I set the disk size to 100GB thinking it >> was enough - but I didn't think it would cause such slowdowns as this, >> considering extracting the Europe PBF takes hours also without the stxxl >> errors. >> >> Server specs: >> Ubuntu 14.04 >> Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) >> 64 GB RAM @ 2133 MHz >> 2 TB Western Digital Enterprise 7200 RPM hard drive >> >> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). >> >> Anyone have any ideas as to what might be going on? Or is it normal to take >> this long without an SSD? >> >> Thanks in advance. >> >> Kind regards, >> Kieran Caplice >> >> >> ___ >> OSRM-talk mailing list >> OSRM-talk@openstreetmap.org >> https://lists.openstreetmap.org/listinfo/osrm-talk >> > ___ > OSRM-talk mailing list > OSRM-talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
Re: [OSRM-talk] osrm-extract taking hours to complete
Hi, On 03/02/2016 09:03 PM, Bjorn Madsen wrote: > I added a high speed SSD and pointed the .stxxl towards that to deal > with the swap slowdown. That bought me a drop in processing time from 12 > hrs -> 3-4 hrs. > > osrm@mat4:~/osrm-backend$ cat .stxxl > *disk=/mnt/tmp/stxxl,40,syscall* Since we're discussing this, I always wondered why I got STXXL messages even when running on a 256 GB RAM machine (could it not simply do everything in RAM there). Then tried to have the STXXL file on a ramdisk but failed because the Linux "tmpfs" doesn't support some IOCTL operation or so that STXXL wants to use. I ended up creating a sparse 200 GB file on the ram disk and loopback-mounting that to put an ext4 file system on it, just to enable STXXL to put its swap file there. Surely better approaches exist? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Kieran, I added a high speed SSD and pointed the .stxxl towards that to deal with the swap slowdown. That bought me a drop in processing time from 12 hrs -> 3-4 hrs. osrm@mat4:~/osrm-backend$ cat .stxxl *disk=/mnt/tmp/stxxl,40,syscall* osrm@mat4:~/osrm-backend$ lsblk NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 223.6G 0 disk ├─sda18:10 4G 0 part │ └─md0 9:00 4G 0 raid1 [SWAP] ├─sda28:20 512M 0 part │ └─md1 9:10 511.7M 0 raid1 /boot └─sda38:30 219.1G 0 part └─md2 9:20 219G 0 raid1 / sdb 8:16 0 223.6G 0 disk ├─sdb18:17 0 4G 0 part │ └─md0 9:00 4G 0 raid1 [SWAP] ├─sdb28:18 0 512M 0 part │ └─md1 9:10 511.7M 0 raid1 /boot └─sdb38:19 0 219.1G 0 part └─md2 9:20 219G 0 raid1 / *sdc 8:32 0 447.1G 0 disk * *└─sdc18:33 0 447.1G 0 part /mnt* On Wed, Mar 2, 2016 at 4:51 PM, Kieran Caplicewrote: > Hello, > > I'm currently extracting the planet PBF (~31 GB), and it's been running > for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 > with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 > minutes to do so from a PBF formatted planet", which is making me wonder > why it's taking so long on our server. Below are some example output > messages: > > [info] Parsing finished after 3584.35 seconds > [extractor] Erasing duplicate nodes ... ok, after 319.091s > [extractor] Sorting all nodes ... ok, after 3632.87s > [extractor] Building node id map ... ok, after 2025.29s > [extractor] Confirming/Writing used nodes ... ok, after 1096.24s > [extractor] Sorting edges by start... ok, after 2000.08s > > Some stxxl errors were outputted as I set the disk size to 100GB thinking > it was enough - but I didn't think it would cause such slowdowns as this, > considering extracting the Europe PBF takes hours also without the stxxl > errors. > > Server specs: > Ubuntu 14.04 > Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) > 64 GB RAM @ 2133 MHz > 2 TB Western Digital Enterprise 7200 RPM hard drive > > At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). > > Anyone have any ideas as to what might be going on? Or is it normal to > take this long without an SSD? > > Thanks in advance. > > Kind regards, > Kieran Caplice > > > ___ > OSRM-talk mailing list > OSRM-talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osrm-talk > > -- Bjorn Madsen Director, Global Operations Multi-Agent Technology Ltd. Bessemer Drive, Stevenage, SG1 2DX b...@multiagenttechnology.com Mobile: +44 779 20 307 20 Main Office: +44 1438 310 035 ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Patrick, Just wanted to ping you off-list. I just had a question about your update processhow often if at all do you update your PBF data, and how do you manage the process? We're thinking of just setting a cron job to download and extract the new PBF every weekend to a temporary folder, stop the server, move the new data to the correct location and start the server again. Kind regards, Kieran Caplice On 02/03/16 17:01, Patrick Niklaus wrote: Hey Kieran, there have been a lot of structural changes (e.g. moving code from osrm-prepare into osrm-extract) that probably invalidate that numbers. Also we support 64bit OSM ids now, which sadly uses a lot more disk space. I think stxxl need like 200GB. I think on our setup we have a turn-around of 6 hours for the planet dataset on an SSD setup (car profile, any other profile needs significantly longer). You should probably think about updating your hard drives as this is IO bound. At your current read/write speed it will already take more than an hour to just write 200GB of data once. We scan it at least twice just for pre-processing. Cheers, Patrick On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplicewrote: Hello, I'm currently extracting the planet PBF (~31 GB), and it's been running for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes to do so from a PBF formatted planet", which is making me wonder why it's taking so long on our server. Below are some example output messages: [info] Parsing finished after 3584.35 seconds [extractor] Erasing duplicate nodes ... ok, after 319.091s [extractor] Sorting all nodes ... ok, after 3632.87s [extractor] Building node id map ... ok, after 2025.29s [extractor] Confirming/Writing used nodes ... ok, after 1096.24s [extractor] Sorting edges by start... ok, after 2000.08s Some stxxl errors were outputted as I set the disk size to 100GB thinking it was enough - but I didn't think it would cause such slowdowns as this, considering extracting the Europe PBF takes hours also without the stxxl errors. Server specs: Ubuntu 14.04 Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) 64 GB RAM @ 2133 MHz 2 TB Western Digital Enterprise 7200 RPM hard drive At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). Anyone have any ideas as to what might be going on? Or is it normal to take this long without an SSD? Thanks in advance. Kind regards, Kieran Caplice ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Patrick, That makes sense then. It's obvious the process is just going to take upwards of 8-10 hours for us in that case. Thanks for the help. Kind regards, Kieran Caplice On 02/03/16 17:01, Patrick Niklaus wrote: Hey Kieran, there have been a lot of structural changes (e.g. moving code from osrm-prepare into osrm-extract) that probably invalidate that numbers. Also we support 64bit OSM ids now, which sadly uses a lot more disk space. I think stxxl need like 200GB. I think on our setup we have a turn-around of 6 hours for the planet dataset on an SSD setup (car profile, any other profile needs significantly longer). You should probably think about updating your hard drives as this is IO bound. At your current read/write speed it will already take more than an hour to just write 200GB of data once. We scan it at least twice just for pre-processing. Cheers, Patrick On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplicewrote: Hello, I'm currently extracting the planet PBF (~31 GB), and it's been running for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes to do so from a PBF formatted planet", which is making me wonder why it's taking so long on our server. Below are some example output messages: [info] Parsing finished after 3584.35 seconds [extractor] Erasing duplicate nodes ... ok, after 319.091s [extractor] Sorting all nodes ... ok, after 3632.87s [extractor] Building node id map ... ok, after 2025.29s [extractor] Confirming/Writing used nodes ... ok, after 1096.24s [extractor] Sorting edges by start... ok, after 2000.08s Some stxxl errors were outputted as I set the disk size to 100GB thinking it was enough - but I didn't think it would cause such slowdowns as this, considering extracting the Europe PBF takes hours also without the stxxl errors. Server specs: Ubuntu 14.04 Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) 64 GB RAM @ 2133 MHz 2 TB Western Digital Enterprise 7200 RPM hard drive At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). Anyone have any ideas as to what might be going on? Or is it normal to take this long without an SSD? Thanks in advance. Kind regards, Kieran Caplice ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
Re: [OSRM-talk] osrm-extract taking hours to complete
Hey Kieran, there have been a lot of structural changes (e.g. moving code from osrm-prepare into osrm-extract) that probably invalidate that numbers. Also we support 64bit OSM ids now, which sadly uses a lot more disk space. I think stxxl need like 200GB. I think on our setup we have a turn-around of 6 hours for the planet dataset on an SSD setup (car profile, any other profile needs significantly longer). You should probably think about updating your hard drives as this is IO bound. At your current read/write speed it will already take more than an hour to just write 200GB of data once. We scan it at least twice just for pre-processing. Cheers, Patrick On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplicewrote: > Hello, > > I'm currently extracting the planet PBF (~31 GB), and it's been running for > hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with > 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes > to do so from a PBF formatted planet", which is making me wonder why it's > taking so long on our server. Below are some example output messages: > > [info] Parsing finished after 3584.35 seconds > [extractor] Erasing duplicate nodes ... ok, after 319.091s > [extractor] Sorting all nodes ... ok, after 3632.87s > [extractor] Building node id map ... ok, after 2025.29s > [extractor] Confirming/Writing used nodes ... ok, after 1096.24s > [extractor] Sorting edges by start... ok, after 2000.08s > > Some stxxl errors were outputted as I set the disk size to 100GB thinking it > was enough - but I didn't think it would cause such slowdowns as this, > considering extracting the Europe PBF takes hours also without the stxxl > errors. > > Server specs: > Ubuntu 14.04 > Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) > 64 GB RAM @ 2133 MHz > 2 TB Western Digital Enterprise 7200 RPM hard drive > > At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). > > Anyone have any ideas as to what might be going on? Or is it normal to take > this long without an SSD? > > Thanks in advance. > > Kind regards, > Kieran Caplice > > > ___ > OSRM-talk mailing list > OSRM-talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osrm-talk > ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk
[OSRM-talk] osrm-extract taking hours to complete
Hello, I'm currently extracting the planet PBF (~31 GB), and it's been running for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes to do so from a PBF formatted planet", which is making me wonder why it's taking so long on our server. Below are some example output messages: [info] Parsing finished after 3584.35 seconds [extractor] Erasing duplicate nodes ... ok, after 319.091s [extractor] Sorting all nodes ... ok, after 3632.87s [extractor] Building node id map ... ok, after 2025.29s [extractor] Confirming/Writing used nodes ... ok, after 1096.24s [extractor] Sorting edges by start... ok, after 2000.08s Some stxxl errors were outputted as I set the disk size to 100GB thinking it was enough - but I didn't think it would cause such slowdowns as this, considering extracting the Europe PBF takes hours also without the stxxl errors. Server specs: Ubuntu 14.04 Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT) 64 GB RAM @ 2133 MHz 2 TB Western Digital Enterprise 7200 RPM hard drive At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%). Anyone have any ideas as to what might be going on? Or is it normal to take this long without an SSD? Thanks in advance. Kind regards, Kieran Caplice ___ OSRM-talk mailing list OSRM-talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/osrm-talk