Re: [OSRM-talk] Required RAM for planet

2016-03-08 Thread Daniel Hofmann 
1/ Provide a swapfile (100GB just in case):

> fallocate -l 100G /path/to/swapfile
> chmod 600 /path/to/swapfile
> mkswap /path/to/swapfile
> swapon /path/to/swapfile

Note: this does not write 100 GB of zeros. Instead what it does is
allocating a certain amount of blocks and just setting the 'uninitialized'
flag on them, returning more or less immediately.


2/ Provide a stxxl config file (250GB should be more than enough, demo
server runs this):

> disk=/path/to/stxxl,25,syscall

See: http://stxxl.sourceforge.net/tags/master/install_config.html


Here is some additional explanation:
https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361


Cheers,

Daniel J H

On Tue, Mar 8, 2016 at 10:02 AM, Kieran Caplice 
wrote:

> Hi all,
>
> Just enquiring if anyone has an up-to-date value for the required RAM to
> extract/prepare the planet PBF on OSRM v4.9.1? Below is the output from
> extraction on a 64 GB RAM machine:
>
> [info] Input file: planet-latest.osm.pbf
> [info] Profile: profile.lua
> [info] Threads: 12
> [info] Using script profile.lua
> [STXXL-MSG] STXXL v1.3.1 (release)
> [STXXL-MSG] 1 disks are allocated, total space: 25 MiB
> [info] Parsing in progress..
> [info] input file generated by planet-dump-ng 1.1.3
> [info] timestamp: 2016-02-29T01:59:57Z
> [info] Using turn restrictions
> [info] Found 3 exceptions to turn restrictions:
> [info]   motorcar
> [info]   motor_vehicle
> [info]   vehicle
> [info] Parsing finished after 3630.62 seconds
> [info] Raw input contains 3240515916 nodes, 333181812 ways, and 4046532
> relations, and 0 unknown entities
> [extractor] Sorting used nodes... ok, after 301.697s
> [extractor] Erasing duplicate nodes   ... ok, after 262.325s
> [extractor] Sorting all nodes ... ok, after 3419.92s
> [extractor] Building node id map  ... ok, after 1674.42s
> [extractor] setting number of nodes   ... ok
> [extractor] Confirming/Writing used nodes ... ok, after 837.176s
> [info] Processed 578533637 nodes
> [extractor] Sorting edges by start... ok, after 1919.47s
> [extractor] Setting start coords  ... ok, after 2657.74s
> [extractor] Sorting edges by target   ... ok, after 1894.56s
> [extractor] Computing edge weights... ok, after 2911.29s
> [extractor] Sorting edges by renumbered start ... ok, after 1864.9s
> [extractor] Writing used edges   ... ok, after 557.274s
> [extractor] setting number of edges   ... ok
> [info] Processed 610970822 edges
> [extractor] Sorting used ways ... ok, after 89.8886s
> [extractor] Sorting 491077 restriction. by from... ok, after 0.906043s
> [extractor] Fixing restriction starts ... ok, after 39.6105s
> [extractor] Sorting restrictions. by to  ... ok, after 0.734124s
> [extractor] Fixing restriction ends   ... ok, after 40.6053s
> [info] usable restrictions: 459264
> [extractor] writing street name index ... ok, after 3.3452s
> [info] extraction finished after 23181.9s
> [info] Generating edge-expanded graph representation
> [info]  - 459264 restrictions.
> [info] Importing n = 578533637 nodes
> [info]  - 157152 bollard nodes, 792260 traffic lights
> [info]  and 610970822 edges
> [info] Graph loaded ok and has 610970822 edges
> [warn] std::bad_alloc
>
> From my reading, this is caused by running out of RAM. The only files
> created were:
>
> -rw-r--r-- 1 rootroot 25G Mar  7 18:57 planet-latest.osrm
> -rw-r--r-- 1 rootroot118M Mar  7 19:00
> planet-latest.osrm.names
> -rw-r--r-- 1 rootroot 15M Mar  7 19:00
> planet-latest.osrm.restrictions
> -rw-r--r-- 1 rootroot  20 Mar  7 12:33
> planet-latest.osrm.timestamp
>
> Which obviously means preparing gives the following:
>
> [info] Input file: planet-latest.osrm
> [info] Profile: profile.lua
> [info] Threads: 12
> [info] Loading edge-expanded graph representation
> [info] Opening planet-latest.osrm.ebg
> [warn] [exception] osrm input file misses magic number. Check or reprocess
> the file
>
> Thanks.
>
> Kind regards,
> Kieran Caplice
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] Required RAM for planet

2016-03-08 Thread Björn Semm 
Hi Kieran,


we needed around 88G of RAM, 100G of SWAP and 250G of stxxl.

I guess SWAP and stxxl is a little bit oversized.


BR

Björn




Von: Kieran Caplice <kieran.capl...@temetra.com>
Gesendet: Dienstag, 8. März 2016 10:02
An: Mailing list to discuss Project OSRM
Betreff: [OSRM-talk] Required RAM for planet

Hi all,

Just enquiring if anyone has an up-to-date value for the required RAM to 
extract/prepare the planet PBF on OSRM v4.9.1? Below is the output from 
extraction on a 64 GB RAM machine:

[info] Input file: planet-latest.osm.pbf
[info] Profile: profile.lua
[info] Threads: 12
[info] Using script profile.lua
[STXXL-MSG] STXXL v1.3.1 (release)
[STXXL-MSG] 1 disks are allocated, total space: 25 MiB
[info] Parsing in progress..
[info] input file generated by planet-dump-ng 1.1.3
[info] timestamp: 2016-02-29T01:59:57Z
[info] Using turn restrictions
[info] Found 3 exceptions to turn restrictions:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parsing finished after 3630.62 seconds
[info] Raw input contains 3240515916 nodes, 333181812 ways, and 4046532 
relations, and 0 unknown entities
[extractor] Sorting used nodes... ok, after 301.697s
[extractor] Erasing duplicate nodes   ... ok, after 262.325s
[extractor] Sorting all nodes ... ok, after 3419.92s
[extractor] Building node id map  ... ok, after 1674.42s
[extractor] setting number of nodes   ... ok
[extractor] Confirming/Writing used nodes ... ok, after 837.176s
[info] Processed 578533637 nodes
[extractor] Sorting edges by start... ok, after 1919.47s
[extractor] Setting start coords  ... ok, after 2657.74s
[extractor] Sorting edges by target   ... ok, after 1894.56s
[extractor] Computing edge weights... ok, after 2911.29s
[extractor] Sorting edges by renumbered start ... ok, after 1864.9s
[extractor] Writing used edges   ... ok, after 557.274s
[extractor] setting number of edges   ... ok
[info] Processed 610970822 edges
[extractor] Sorting used ways ... ok, after 89.8886s
[extractor] Sorting 491077 restriction. by from... ok, after 0.906043s
[extractor] Fixing restriction starts ... ok, after 39.6105s
[extractor] Sorting restrictions. by to  ... ok, after 0.734124s
[extractor] Fixing restriction ends   ... ok, after 40.6053s
[info] usable restrictions: 459264
[extractor] writing street name index ... ok, after 3.3452s
[info] extraction finished after 23181.9s
[info] Generating edge-expanded graph representation
[info]  - 459264 restrictions.
[info] Importing n = 578533637 nodes
[info]  - 157152 bollard nodes, 792260 traffic lights
[info]  and 610970822 edges
[info] Graph loaded ok and has 610970822 edges
[warn] std::bad_alloc

>From my reading, this is caused by running out of RAM. The only files created 
>were:

-rw-r--r-- 1 rootroot 25G Mar  7 18:57 planet-latest.osrm
-rw-r--r-- 1 rootroot118M Mar  7 19:00 planet-latest.osrm.names
-rw-r--r-- 1 rootroot 15M Mar  7 19:00 
planet-latest.osrm.restrictions
-rw-r--r-- 1 rootroot  20 Mar  7 12:33 
planet-latest.osrm.timestamp

Which obviously means preparing gives the following:

[info] Input file: planet-latest.osrm
[info] Profile: profile.lua
[info] Threads: 12
[info] Loading edge-expanded graph representation
[info] Opening planet-latest.osrm.ebg
[warn] [exception] osrm input file misses magic number. Check or reprocess the 
file

Thanks.

Kind regards,
Kieran Caplice

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

[OSRM-talk] Required RAM for planet

2016-03-08 Thread Kieran Caplice 

Hi all,

Just enquiring if anyone has an up-to-date value for the required RAM to 
extract/prepare the planet PBF on OSRM v4.9.1? Below is the output from 
extraction on a 64 GB RAM machine:


[info] Input file: planet-latest.osm.pbf
[info] Profile: profile.lua
[info] Threads: 12
[info] Using script profile.lua
[STXXL-MSG] STXXL v1.3.1 (release)
[STXXL-MSG] 1 disks are allocated, total space: 25 MiB
[info] Parsing in progress..
[info] input file generated by planet-dump-ng 1.1.3
[info] timestamp: 2016-02-29T01:59:57Z
[info] Using turn restrictions
[info] Found 3 exceptions to turn restrictions:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parsing finished after 3630.62 seconds
[info] Raw input contains 3240515916 nodes, 333181812 ways, and 4046532 
relations, and 0 unknown entities

[extractor] Sorting used nodes... ok, after 301.697s
[extractor] Erasing duplicate nodes   ... ok, after 262.325s
[extractor] Sorting all nodes ... ok, after 3419.92s
[extractor] Building node id map  ... ok, after 1674.42s
[extractor] setting number of nodes   ... ok
[extractor] Confirming/Writing used nodes ... ok, after 837.176s
[info] Processed 578533637 nodes
[extractor] Sorting edges by start... ok, after 1919.47s
[extractor] Setting start coords  ... ok, after 2657.74s
[extractor] Sorting edges by target   ... ok, after 1894.56s
[extractor] Computing edge weights... ok, after 2911.29s
[extractor] Sorting edges by renumbered start ... ok, after 1864.9s
[extractor] Writing used edges   ... ok, after 557.274s
[extractor] setting number of edges   ... ok
[info] Processed 610970822 edges
[extractor] Sorting used ways ... ok, after 89.8886s
[extractor] Sorting 491077 restriction. by from... ok, after 0.906043s
[extractor] Fixing restriction starts ... ok, after 39.6105s
[extractor] Sorting restrictions. by to  ... ok, after 0.734124s
[extractor] Fixing restriction ends   ... ok, after 40.6053s
[info] usable restrictions: 459264
[extractor] writing street name index ... ok, after 3.3452s
[info] extraction finished after 23181.9s
[info] Generating edge-expanded graph representation
[info]  - 459264 restrictions.
[info] Importing n = 578533637 nodes
[info]  - 157152 bollard nodes, 792260 traffic lights
[info]  and 610970822 edges
[info] Graph loaded ok and has 610970822 edges
[warn] std::bad_alloc

From my reading, this is caused by running out of RAM. The only files 
created were:


-rw-r--r-- 1 rootroot 25G Mar  7 18:57 planet-latest.osrm
-rw-r--r-- 1 rootroot118M Mar  7 19:00 
planet-latest.osrm.names
-rw-r--r-- 1 rootroot 15M Mar  7 19:00 
planet-latest.osrm.restrictions
-rw-r--r-- 1 rootroot  20 Mar  7 12:33 
planet-latest.osrm.timestamp


Which obviously means preparing gives the following:

[info] Input file: planet-latest.osrm
[info] Profile: profile.lua
[info] Threads: 12
[info] Loading edge-expanded graph representation
[info] Opening planet-latest.osrm.ebg
[warn] [exception] osrm input file misses magic number. Check or 
reprocess the file


Thanks.

Kind regards,
Kieran Caplice

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk