Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Daniel Patterson]

OSRM pre-processing is kind of worst-case for disk swapping - it does a lot
of random access, so if you run out of RAM, pages will get continuously
swapped in/out.  The slowdown between RAM-only and swap-on-SSD can be 50x
or more.  The difference between swap on a spinning disk and an SSD is a
good 10x on top of that.

If you can, avoid letting OSRM pre-processing touch swap at all costs -
it's worth renting a big machine on AWS/Google/Azure for a short time to
pre-process the data.

daniel

On Fri, Sep 22, 2017 at 7:14 AM, Sayer, Bryan <bsa...@s-3.com> wrote:

> Just as a point of reference, you mention that avoiding swapping and just
> using RAM will be much faster. If the swap space is an SSD drive, how much
> does using only RAM speed up contract? That is, what is the difference
> between using say a 7200 rpm SATA drive versus an SSD drive make?
> --
> *From:* Daniel Patterson <dan...@mapbox.com>
> *Sent:* Friday, September 22, 2017 8:05:56 AM
> *To:* Mailing list to discuss Project OSRM
> *Subject:* Re: [OSRM-talk] Expecting time for osrm-contract for planet
>
> Hi Kieran,
>
>   We don't, but I've done some profiling in the past, Docker itself should
> have negligible impact.
>
>   The two things that might be slowing things down:
>
> 1) The docker images don't bundle jemalloc (http://jemalloc.net/),
> which can speed things up by 10-15%
> 2) osrm-contract is very CPU intensive - the more threads you give it,
> the faster it will run.  Give it more if you can.
>
>  256GB of RAM should be more than enough to avoid swapping, but you might
> want to disable swap to be sure - if it's being used, it will have a huge
> impact on speed.
>
> daniel
>
> On Fri, Sep 22, 2017 at 2:30 AM, Kieran Caplice <
> kieran.capl...@temetra.com> wrote:
>
>> Hi Daniel,
>> Can you clarify if you use Docker at Mapbox? What kind of server do you
>> guys have (if not the one described on the wiki page)?
>>
>> Over night, the process has gone to just 75% complete from 65% yesterday
>> - 41 hours in total now, and back to maxing CPU again.
>>
>> Kind regards,
>> Kieran Caplice
>>
>> On 21/09/17 17:17, Daniel Patterson wrote:
>>
>> OSRM supports *two* core routing algorithms - CH and MLD.  The
>> `osrm-contract` tool generates the CH dataset, but you can use the MLD
>> pipeline instead with:
>>
>> osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf
>> osrm-partition yourmap.osrm
>> osrm-customize yourmap.osrm
>> osrm-routed -a MLD yourmap.osrm
>>
>> This sequence of tools should be significantly quicker than osrm-contract
>> - the price you pay is that routing requests are about 5x slower (still
>> pretty fast though!).  The reason that MLD exists is for the
>> `osrm-customize` step - it allows you to import traffic data very quickly
>> and update the routing graph (~1 minute for North America).
>>
>> It's hard to say exactly what's going wrong with osrm-contract here -
>> here at Mapbox, we daily run `osrm-contract` over the latest planet with
>> the bicycle profile without a problem, however, Alex and others have
>> reported issues with what seem like hangs on much smaller datasets that
>> we've been unable to reproduce so far.
>>
>> The runtime of osrm-contract is affected by how much hierarchy exists in
>> the data - the more similar the edge speeds (like in foot) and the more
>> edges there are, the slower it gets, often in a non-linear fashion.  The
>> car profile has a very hierarchical structure (many different road speeds),
>> so it fits well into the CH, the construction algorithm  doesn't need to
>> compare as many options.
>>
>> daniel
>>
>> On Thu, Sep 21, 2017 at 9:03 AM, Kieran Caplice <
>> kieran.capl...@temetra.com> wrote:
>>
>>> We're actually looking for the best of both car and foot, so in my head,
>>> bicycle would be the happy medium (though I could be completely wrong on
>>> this).
>>> Kind regards,
>>> Kieran Caplice
>>>
>>> On 21/09/17 16:53, Alex Farioletti wrote:
>>>
>>> i've run into the same issues, and now i just use metroextracts of the
>>> areas that i need for the bike stuff i do and it reduces the time
>>> significantly
>>>
>>> *Alex Farioletti*
>>> *415.312.1674*
>>> *tcbcourier.com <http://tcbcourier.com> *
>>>
>>> On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice <
>>> kieran.capl...@temetra.com> wrote:
>>>
>>>> Thanks Daniel.
>>>>
>>>> I'm using the bicycle p

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Sayer, Bryan]

Just as a point of reference, you mention that avoiding swapping and just using 
RAM will be much faster. If the swap space is an SSD drive, how much does using 
only RAM speed up contract? That is, what is the difference between using say a 
7200 rpm SATA drive versus an SSD drive make?


From: Daniel Patterson <dan...@mapbox.com>
Sent: Friday, September 22, 2017 8:05:56 AM
To: Mailing list to discuss Project OSRM
Subject: Re: [OSRM-talk] Expecting time for osrm-contract for planet

Hi Kieran,

  We don't, but I've done some profiling in the past, Docker itself should have 
negligible impact.

  The two things that might be slowing things down:

1) The docker images don't bundle jemalloc (http://jemalloc.net/), which 
can speed things up by 10-15%
2) osrm-contract is very CPU intensive - the more threads you give it, the 
faster it will run.  Give it more if you can.

 256GB of RAM should be more than enough to avoid swapping, but you might want 
to disable swap to be sure - if it's being used, it will have a huge impact on 
speed.

daniel

On Fri, Sep 22, 2017 at 2:30 AM, Kieran Caplice 
<kieran.capl...@temetra.com<mailto:kieran.capl...@temetra.com>> wrote:

Hi Daniel,

Can you clarify if you use Docker at Mapbox? What kind of server do you guys 
have (if not the one described on the wiki page)?

Over night, the process has gone to just 75% complete from 65% yesterday - 41 
hours in total now, and back to maxing CPU again.

Kind regards,
Kieran Caplice

On 21/09/17 17:17, Daniel Patterson wrote:
OSRM supports *two* core routing algorithms - CH and MLD.  The `osrm-contract` 
tool generates the CH dataset, but you can use the MLD pipeline instead with:

osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf
osrm-partition yourmap.osrm
osrm-customize yourmap.osrm
osrm-routed -a MLD yourmap.osrm

This sequence of tools should be significantly quicker than osrm-contract - the 
price you pay is that routing requests are about 5x slower (still pretty fast 
though!).  The reason that MLD exists is for the `osrm-customize` step - it 
allows you to import traffic data very quickly and update the routing graph (~1 
minute for North America).

It's hard to say exactly what's going wrong with osrm-contract here - here at 
Mapbox, we daily run `osrm-contract` over the latest planet with the bicycle 
profile without a problem, however, Alex and others have reported issues with 
what seem like hangs on much smaller datasets that we've been unable to 
reproduce so far.

The runtime of osrm-contract is affected by how much hierarchy exists in the 
data - the more similar the edge speeds (like in foot) and the more edges there 
are, the slower it gets, often in a non-linear fashion.  The car profile has a 
very hierarchical structure (many different road speeds), so it fits well into 
the CH, the construction algorithm  doesn't need to compare as many options.

daniel

On Thu, Sep 21, 2017 at 9:03 AM, Kieran Caplice 
<kieran.capl...@temetra.com<mailto:kieran.capl...@temetra.com>> wrote:

We're actually looking for the best of both car and foot, so in my head, 
bicycle would be the happy medium (though I could be completely wrong on this).

Kind regards,
Kieran Caplice

On 21/09/17 16:53, Alex Farioletti wrote:
i've run into the same issues, and now i just use metroextracts of the areas 
that i need for the bike stuff i do and it reduces the time significantly

Alex Farioletti
415.312.1674
tcbcourier.com<http://tcbcourier.com>

On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice 
<kieran.capl...@temetra.com<mailto:kieran.capl...@temetra.com>> wrote:

Thanks Daniel.

I'm using the bicycle profile, so I would expect based on what you've said that 
somewhere up to 36 hours would be likely. However, this is the current output, 
after 25h40m:

[info] Input file: /data/1505492056/planet-latest.osrm
[info] Threads: 12
[info] Reading node weights.
[info] Done reading node weights.
[info] Loading edge-expanded graph representation
[info] merged 2379332 edges out of 152432
[info] initializing node priorities... ok.
[info] preprocessing 389797971 (90%) nodes...
[info] . 10% . 20% . 30% . 40% . 50% . 60%

It hasn't advanced past 60% in the last 2-3 hours. It is however maxing CPU and 
using approximately the same amount of RAM since it started.

Kind regards,
Kieran Caplice

On 21/09/17 16:39, Daniel Patterson wrote:
Hi Kieran,

  The contraction time will be slow - many, many hours for the whole planet.  
*Typically* for the car profile it's about 12 hours, but if you use bike or 
foot, or your own profile, it can get a lot bigger.

  If you've messed with the travel speeds, that can have a big effect too.  24 
hours is not unheard of, but whether it's legit will depend a lot on the 
details.

daniel

On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice 
<kieran.capl...@temetra.com<mailto:kieran.capl...@temetra.com>> wrote:
Hi all,

Could

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Daniel Patterson]

Hi Kieran,

  We don't, but I've done some profiling in the past, Docker itself should
have negligible impact.

  The two things that might be slowing things down:

1) The docker images don't bundle jemalloc (http://jemalloc.net/),
which can speed things up by 10-15%
2) osrm-contract is very CPU intensive - the more threads you give it,
the faster it will run.  Give it more if you can.

 256GB of RAM should be more than enough to avoid swapping, but you might
want to disable swap to be sure - if it's being used, it will have a huge
impact on speed.

daniel

On Fri, Sep 22, 2017 at 2:30 AM, Kieran Caplice 
wrote:

> Hi Daniel,
> Can you clarify if you use Docker at Mapbox? What kind of server do you
> guys have (if not the one described on the wiki page)?
>
> Over night, the process has gone to just 75% complete from 65% yesterday -
> 41 hours in total now, and back to maxing CPU again.
>
> Kind regards,
> Kieran Caplice
>
> On 21/09/17 17:17, Daniel Patterson wrote:
>
> OSRM supports *two* core routing algorithms - CH and MLD.  The
> `osrm-contract` tool generates the CH dataset, but you can use the MLD
> pipeline instead with:
>
> osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf
> osrm-partition yourmap.osrm
> osrm-customize yourmap.osrm
> osrm-routed -a MLD yourmap.osrm
>
> This sequence of tools should be significantly quicker than osrm-contract
> - the price you pay is that routing requests are about 5x slower (still
> pretty fast though!).  The reason that MLD exists is for the
> `osrm-customize` step - it allows you to import traffic data very quickly
> and update the routing graph (~1 minute for North America).
>
> It's hard to say exactly what's going wrong with osrm-contract here - here
> at Mapbox, we daily run `osrm-contract` over the latest planet with the
> bicycle profile without a problem, however, Alex and others have reported
> issues with what seem like hangs on much smaller datasets that we've been
> unable to reproduce so far.
>
> The runtime of osrm-contract is affected by how much hierarchy exists in
> the data - the more similar the edge speeds (like in foot) and the more
> edges there are, the slower it gets, often in a non-linear fashion.  The
> car profile has a very hierarchical structure (many different road speeds),
> so it fits well into the CH, the construction algorithm  doesn't need to
> compare as many options.
>
> daniel
>
> On Thu, Sep 21, 2017 at 9:03 AM, Kieran Caplice <
> kieran.capl...@temetra.com> wrote:
>
>> We're actually looking for the best of both car and foot, so in my head,
>> bicycle would be the happy medium (though I could be completely wrong on
>> this).
>> Kind regards,
>> Kieran Caplice
>>
>> On 21/09/17 16:53, Alex Farioletti wrote:
>>
>> i've run into the same issues, and now i just use metroextracts of the
>> areas that i need for the bike stuff i do and it reduces the time
>> significantly
>>
>> *Alex Farioletti*
>> *415.312.1674*
>> *tcbcourier.com  *
>>
>> On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice <
>> kieran.capl...@temetra.com> wrote:
>>
>>> Thanks Daniel.
>>>
>>> I'm using the bicycle profile, so I would expect based on what you've
>>> said that somewhere up to 36 hours would be likely. However, this is the
>>> current output, after 25h40m:
>>> [info] Input file: /data/1505492056/planet-latest.osrm
>>> [info] Threads: 12
>>> [info] Reading node weights.
>>> [info] Done reading node weights.
>>> [info] Loading edge-expanded graph representation
>>> [info] merged 2379332 edges out of 152432
>>> [info] initializing node priorities... ok.
>>> [info] preprocessing 389797971 (90%) nodes...
>>> [info] . 10% . 20% . 30% . 40% . 50% . 60%
>>>
>>> It hasn't advanced past 60% in the last 2-3 hours. It is however maxing
>>> CPU and using approximately the same amount of RAM since it started.
>>>
>>> Kind regards,
>>> Kieran Caplice
>>>
>>> On 21/09/17 16:39, Daniel Patterson wrote:
>>>
>>> Hi Kieran,
>>>
>>>   The contraction time will be slow - many, many hours for the whole
>>> planet.  *Typically* for the car profile it's about 12 hours, but if you
>>> use bike or foot, or your own profile, it can get a lot bigger.
>>>
>>>   If you've messed with the travel speeds, that can have a big effect
>>> too.  24 hours is not unheard of, but whether it's legit will depend a lot
>>> on the details.
>>>
>>> daniel
>>>
>>> On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice <
>>> kieran.capl...@temetra.com> wrote:
>>>
 Hi all,

 Could anyone give an approx estimate for the time required to run the
 osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD machine?

 The osrm-extract process finished in 232 minutes, but the contract has
 now been running solid for 24 hours, and appears to be stuck at 60% on
 "preprocessing nodes". All 12 cores are generally maxed out, and the
 process is using nearly 90 GB of RAM.

 This is the second time I've run 

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Kieran Caplice]

Hi Daniel,

Can you clarify if you use Docker at Mapbox? What kind of server do you 
guys have (if not the one described on the wiki page)?


Over night, the process has gone to just 75% complete from 65% yesterday 
- 41 hours in total now, and back to maxing CPU again.


Kind regards,
Kieran Caplice

On 21/09/17 17:17, Daniel Patterson wrote:
OSRM supports *two* core routing algorithms - CH and MLD.  The 
`osrm-contract` tool generates the CH dataset, but you can use the MLD 
pipeline instead with:


osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf
osrm-partition yourmap.osrm
osrm-customize yourmap.osrm
osrm-routed -a MLD yourmap.osrm

This sequence of tools should be significantly quicker than 
osrm-contract - the price you pay is that routing requests are about 
5x slower (still pretty fast though!).  The reason that MLD exists is 
for the `osrm-customize` step - it allows you to import traffic data 
very quickly and update the routing graph (~1 minute for North America).


It's hard to say exactly what's going wrong with osrm-contract here - 
here at Mapbox, we daily run `osrm-contract` over the latest planet 
with the bicycle profile without a problem, however, Alex and others 
have reported issues with what seem like hangs on much smaller 
datasets that we've been unable to reproduce so far.


The runtime of osrm-contract is affected by how much hierarchy exists 
in the data - the more similar the edge speeds (like in foot) and the 
more edges there are, the slower it gets, often in a non-linear 
fashion.  The car profile has a very hierarchical structure (many 
different road speeds), so it fits well into the CH, the construction 
algorithm  doesn't need to compare as many options.


daniel

On Thu, Sep 21, 2017 at 9:03 AM, Kieran Caplice 
> wrote:


We're actually looking for the best of both car and foot, so in my
head, bicycle would be the happy medium (though I could be
completely wrong on this).

Kind regards,
Kieran Caplice

On 21/09/17 16:53, Alex Farioletti wrote:

i've run into the same issues, and now i just use metroextracts
of the areas that i need for the bike stuff i do and it reduces
the time significantly

*Alex Farioletti*
*415.312.1674*
/tcbcourier.com  /

On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice
>
wrote:

Thanks Daniel.

I'm using the bicycle profile, so I would expect based on
what you've said that somewhere up to 36 hours would be
likely. However, this is the current output, after 25h40m:

[info] Input file: /data/1505492056/planet-latest.osrm
[info] Threads: 12
[info] Reading node weights.
[info] Done reading node weights.
[info] Loading edge-expanded graph representation
[info] merged 2379332 edges out of 152432
[info] initializing node priorities... ok.
[info] preprocessing 389797971 (90%) nodes...
[info] . 10% . 20% . 30% . 40% . 50% . 60%

It hasn't advanced past 60% in the last 2-3 hours. It is
however maxing CPU and using approximately the same amount of
RAM since it started.

Kind regards,
Kieran Caplice

On 21/09/17 16:39, Daniel Patterson wrote:

Hi Kieran,

  The contraction time will be slow - many, many hours for
the whole planet.  *Typically* for the car profile it's
about 12 hours, but if you use bike or foot, or your own
profile, it can get a lot bigger.

  If you've messed with the travel speeds, that can have a
big effect too.  24 hours is not unheard of, but whether
it's legit will depend a lot on the details.

daniel

On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice
> wrote:

Hi all,

Could anyone give an approx estimate for the time
required to run the osrm-contract on planet data on a 12
thread, 256 GB RAM, SSD machine?

The osrm-extract process finished in 232 minutes, but
the contract has now been running solid for 24 hours,
and appears to be stuck at 60% on "preprocessing nodes".
All 12 cores are generally maxed out, and the process is
using nearly 90 GB of RAM.

This is the second time I've run the contract process,
as my SSH connection to the server dropped the first
time and the process wasn't running in a screen etc, so
I assumed after the 40-odd hours it was running for, the
connection drop caused it to hang, but now I'm not so
sure. Were there any files I should maybe have cleared
before trying to run it again?

I'm using 

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Daniel Patterson]

OSRM supports *two* core routing algorithms - CH and MLD.  The
`osrm-contract` tool generates the CH dataset, but you can use the MLD
pipeline instead with:

osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf
osrm-partition yourmap.osrm
osrm-customize yourmap.osrm
osrm-routed -a MLD yourmap.osrm

This sequence of tools should be significantly quicker than osrm-contract -
the price you pay is that routing requests are about 5x slower (still
pretty fast though!).  The reason that MLD exists is for the
`osrm-customize` step - it allows you to import traffic data very quickly
and update the routing graph (~1 minute for North America).

It's hard to say exactly what's going wrong with osrm-contract here - here
at Mapbox, we daily run `osrm-contract` over the latest planet with the
bicycle profile without a problem, however, Alex and others have reported
issues with what seem like hangs on much smaller datasets that we've been
unable to reproduce so far.

The runtime of osrm-contract is affected by how much hierarchy exists in
the data - the more similar the edge speeds (like in foot) and the more
edges there are, the slower it gets, often in a non-linear fashion.  The
car profile has a very hierarchical structure (many different road speeds),
so it fits well into the CH, the construction algorithm  doesn't need to
compare as many options.

daniel

On Thu, Sep 21, 2017 at 9:03 AM, Kieran Caplice 
wrote:

> We're actually looking for the best of both car and foot, so in my head,
> bicycle would be the happy medium (though I could be completely wrong on
> this).
> Kind regards,
> Kieran Caplice
>
> On 21/09/17 16:53, Alex Farioletti wrote:
>
> i've run into the same issues, and now i just use metroextracts of the
> areas that i need for the bike stuff i do and it reduces the time
> significantly
>
> *Alex Farioletti*
> *415.312.1674*
> *tcbcourier.com  *
>
> On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice <
> kieran.capl...@temetra.com> wrote:
>
>> Thanks Daniel.
>>
>> I'm using the bicycle profile, so I would expect based on what you've
>> said that somewhere up to 36 hours would be likely. However, this is the
>> current output, after 25h40m:
>> [info] Input file: /data/1505492056/planet-latest.osrm
>> [info] Threads: 12
>> [info] Reading node weights.
>> [info] Done reading node weights.
>> [info] Loading edge-expanded graph representation
>> [info] merged 2379332 edges out of 152432
>> [info] initializing node priorities... ok.
>> [info] preprocessing 389797971 (90%) nodes...
>> [info] . 10% . 20% . 30% . 40% . 50% . 60%
>>
>> It hasn't advanced past 60% in the last 2-3 hours. It is however maxing
>> CPU and using approximately the same amount of RAM since it started.
>>
>> Kind regards,
>> Kieran Caplice
>>
>> On 21/09/17 16:39, Daniel Patterson wrote:
>>
>> Hi Kieran,
>>
>>   The contraction time will be slow - many, many hours for the whole
>> planet.  *Typically* for the car profile it's about 12 hours, but if you
>> use bike or foot, or your own profile, it can get a lot bigger.
>>
>>   If you've messed with the travel speeds, that can have a big effect
>> too.  24 hours is not unheard of, but whether it's legit will depend a lot
>> on the details.
>>
>> daniel
>>
>> On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice <
>> kieran.capl...@temetra.com> wrote:
>>
>>> Hi all,
>>>
>>> Could anyone give an approx estimate for the time required to run the
>>> osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD machine?
>>>
>>> The osrm-extract process finished in 232 minutes, but the contract has
>>> now been running solid for 24 hours, and appears to be stuck at 60% on
>>> "preprocessing nodes". All 12 cores are generally maxed out, and the
>>> process is using nearly 90 GB of RAM.
>>>
>>> This is the second time I've run the contract process, as my SSH
>>> connection to the server dropped the first time and the process wasn't
>>> running in a screen etc, so I assumed after the 40-odd hours it was running
>>> for, the connection drop caused it to hang, but now I'm not so sure. Were
>>> there any files I should maybe have cleared before trying to run it again?
>>>
>>> I'm using the docker image to run the command (using
>>> osrm/osrm-backend:latest): time docker run -t -v /opt/osrm/data:/data
>>> osrm/osrm-backend osrm-contract /data/1505492056/planet-latest.osrm
>>>
>>> Kind regards,
>>> Kieran Caplice
>>>
>>>
>>> ___
>>> OSRM-talk mailing list
>>> OSRM-talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>>
>>
>>
>>
>> ___
>> OSRM-talk mailing 
>> listOSRM-talk@openstreetmap.orghttps://lists.openstreetmap.org/listinfo/osrm-talk
>>
>>
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>>
>
>
> 

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Kieran Caplice]

I take back what I said, the contract process has advanced 5%! Looks 
like it's now using more RAM than CPU, so I'll give it a bit more time 
before judging it prematurely again :-)


Kind regards,
Kieran Caplice

On 21/09/17 17:03, Kieran Caplice wrote:


We're actually looking for the best of both car and foot, so in my 
head, bicycle would be the happy medium (though I could be completely 
wrong on this).


Kind regards,
Kieran Caplice

On 21/09/17 16:53, Alex Farioletti wrote:
i've run into the same issues, and now i just use metroextracts of 
the areas that i need for the bike stuff i do and it reduces the time 
significantly


*Alex Farioletti*
*415.312.1674*
/tcbcourier.com  /

On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice 
> wrote:


Thanks Daniel.

I'm using the bicycle profile, so I would expect based on what
you've said that somewhere up to 36 hours would be likely.
However, this is the current output, after 25h40m:

[info] Input file: /data/1505492056/planet-latest.osrm
[info] Threads: 12
[info] Reading node weights.
[info] Done reading node weights.
[info] Loading edge-expanded graph representation
[info] merged 2379332 edges out of 152432
[info] initializing node priorities... ok.
[info] preprocessing 389797971 (90%) nodes...
[info] . 10% . 20% . 30% . 40% . 50% . 60%

It hasn't advanced past 60% in the last 2-3 hours. It is however
maxing CPU and using approximately the same amount of RAM since
it started.

Kind regards,
Kieran Caplice

On 21/09/17 16:39, Daniel Patterson wrote:

Hi Kieran,

  The contraction time will be slow - many, many hours for the
whole planet.  *Typically* for the car profile it's about 12
hours, but if you use bike or foot, or your own profile, it can
get a lot bigger.

  If you've messed with the travel speeds, that can have a big
effect too.  24 hours is not unheard of, but whether it's legit
will depend a lot on the details.

daniel

On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice
>
wrote:

Hi all,

Could anyone give an approx estimate for the time required
to run the osrm-contract on planet data on a 12 thread, 256
GB RAM, SSD machine?

The osrm-extract process finished in 232 minutes, but the
contract has now been running solid for 24 hours, and
appears to be stuck at 60% on "preprocessing nodes". All 12
cores are generally maxed out, and the process is using
nearly 90 GB of RAM.

This is the second time I've run the contract process, as my
SSH connection to the server dropped the first time and the
process wasn't running in a screen etc, so I assumed after
the 40-odd hours it was running for, the connection drop
caused it to hang, but now I'm not so sure. Were there any
files I should maybe have cleared before trying to run it again?

I'm using the docker image to run the command (using
osrm/osrm-backend:latest): time docker run -t -v
/opt/osrm/data:/data osrm/osrm-backend osrm-contract
/data/1505492056/planet-latest.osrm

Kind regards,
Kieran Caplice


___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/osrm-talk





___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/osrm-talk




___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/osrm-talk





___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk




___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk


___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Alex Farioletti]

i've run into the same issues, and now i just use metroextracts of the
areas that i need for the bike stuff i do and it reduces the time
significantly

*Alex Farioletti*
*415.312.1674*
*tcbcourier.com  *

On Thu, Sep 21, 2017 at 8:49 AM, Kieran Caplice 
wrote:

> Thanks Daniel.
>
> I'm using the bicycle profile, so I would expect based on what you've said
> that somewhere up to 36 hours would be likely. However, this is the current
> output, after 25h40m:
> [info] Input file: /data/1505492056/planet-latest.osrm
> [info] Threads: 12
> [info] Reading node weights.
> [info] Done reading node weights.
> [info] Loading edge-expanded graph representation
> [info] merged 2379332 edges out of 152432
> [info] initializing node priorities... ok.
> [info] preprocessing 389797971 (90%) nodes...
> [info] . 10% . 20% . 30% . 40% . 50% . 60%
>
> It hasn't advanced past 60% in the last 2-3 hours. It is however maxing
> CPU and using approximately the same amount of RAM since it started.
>
> Kind regards,
> Kieran Caplice
>
> On 21/09/17 16:39, Daniel Patterson wrote:
>
> Hi Kieran,
>
>   The contraction time will be slow - many, many hours for the whole
> planet.  *Typically* for the car profile it's about 12 hours, but if you
> use bike or foot, or your own profile, it can get a lot bigger.
>
>   If you've messed with the travel speeds, that can have a big effect too.
>  24 hours is not unheard of, but whether it's legit will depend a lot on
> the details.
>
> daniel
>
> On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice <
> kieran.capl...@temetra.com> wrote:
>
>> Hi all,
>>
>> Could anyone give an approx estimate for the time required to run the
>> osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD machine?
>>
>> The osrm-extract process finished in 232 minutes, but the contract has
>> now been running solid for 24 hours, and appears to be stuck at 60% on
>> "preprocessing nodes". All 12 cores are generally maxed out, and the
>> process is using nearly 90 GB of RAM.
>>
>> This is the second time I've run the contract process, as my SSH
>> connection to the server dropped the first time and the process wasn't
>> running in a screen etc, so I assumed after the 40-odd hours it was running
>> for, the connection drop caused it to hang, but now I'm not so sure. Were
>> there any files I should maybe have cleared before trying to run it again?
>>
>> I'm using the docker image to run the command (using
>> osrm/osrm-backend:latest): time docker run -t -v /opt/osrm/data:/data
>> osrm/osrm-backend osrm-contract /data/1505492056/planet-latest.osrm
>>
>> Kind regards,
>> Kieran Caplice
>>
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>
>
>
> ___
> OSRM-talk mailing 
> listOSRM-talk@openstreetmap.orghttps://lists.openstreetmap.org/listinfo/osrm-talk
>
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Kieran Caplice]

Thanks Daniel.

I'm using the bicycle profile, so I would expect based on what you've 
said that somewhere up to 36 hours would be likely. However, this is the 
current output, after 25h40m:


[info] Input file: /data/1505492056/planet-latest.osrm
[info] Threads: 12
[info] Reading node weights.
[info] Done reading node weights.
[info] Loading edge-expanded graph representation
[info] merged 2379332 edges out of 152432
[info] initializing node priorities... ok.
[info] preprocessing 389797971 (90%) nodes...
[info] . 10% . 20% . 30% . 40% . 50% . 60%

It hasn't advanced past 60% in the last 2-3 hours. It is however maxing 
CPU and using approximately the same amount of RAM since it started.


Kind regards,
Kieran Caplice

On 21/09/17 16:39, Daniel Patterson wrote:

Hi Kieran,

  The contraction time will be slow - many, many hours for the whole 
planet.  *Typically* for the car profile it's about 12 hours, but if 
you use bike or foot, or your own profile, it can get a lot bigger.


  If you've messed with the travel speeds, that can have a big effect 
too.  24 hours is not unheard of, but whether it's legit will depend a 
lot on the details.


daniel

On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice 
> wrote:


Hi all,

Could anyone give an approx estimate for the time required to run
the osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD
machine?

The osrm-extract process finished in 232 minutes, but the contract
has now been running solid for 24 hours, and appears to be stuck
at 60% on "preprocessing nodes". All 12 cores are generally maxed
out, and the process is using nearly 90 GB of RAM.

This is the second time I've run the contract process, as my SSH
connection to the server dropped the first time and the process
wasn't running in a screen etc, so I assumed after the 40-odd
hours it was running for, the connection drop caused it to hang,
but now I'm not so sure. Were there any files I should maybe have
cleared before trying to run it again?

I'm using the docker image to run the command (using
osrm/osrm-backend:latest): time docker run -t -v
/opt/osrm/data:/data osrm/osrm-backend osrm-contract
/data/1505492056/planet-latest.osrm

Kind regards,
Kieran Caplice


___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/osrm-talk





___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk


___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] Expecting time for osrm-contract for planet

[Daniel Patterson]

Hi Kieran,

  The contraction time will be slow - many, many hours for the whole
planet.  *Typically* for the car profile it's about 12 hours, but if you
use bike or foot, or your own profile, it can get a lot bigger.

  If you've messed with the travel speeds, that can have a big effect too.
 24 hours is not unheard of, but whether it's legit will depend a lot on
the details.

daniel

On Thu, Sep 21, 2017 at 7:00 AM, Kieran Caplice 
wrote:

> Hi all,
>
> Could anyone give an approx estimate for the time required to run the
> osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD machine?
>
> The osrm-extract process finished in 232 minutes, but the contract has now
> been running solid for 24 hours, and appears to be stuck at 60% on
> "preprocessing nodes". All 12 cores are generally maxed out, and the
> process is using nearly 90 GB of RAM.
>
> This is the second time I've run the contract process, as my SSH
> connection to the server dropped the first time and the process wasn't
> running in a screen etc, so I assumed after the 40-odd hours it was running
> for, the connection drop caused it to hang, but now I'm not so sure. Were
> there any files I should maybe have cleared before trying to run it again?
>
> I'm using the docker image to run the command (using
> osrm/osrm-backend:latest): time docker run -t -v /opt/osrm/data:/data
> osrm/osrm-backend osrm-contract /data/1505492056/planet-latest.osrm
>
> Kind regards,
> Kieran Caplice
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

[OSRM-talk] Expecting time for osrm-contract for planet

[Kieran Caplice]

Hi all,

Could anyone give an approx estimate for the time required to run the 
osrm-contract on planet data on a 12 thread, 256 GB RAM, SSD machine?


The osrm-extract process finished in 232 minutes, but the contract has 
now been running solid for 24 hours, and appears to be stuck at 60% on 
"preprocessing nodes". All 12 cores are generally maxed out, and the 
process is using nearly 90 GB of RAM.


This is the second time I've run the contract process, as my SSH 
connection to the server dropped the first time and the process wasn't 
running in a screen etc, so I assumed after the 40-odd hours it was 
running for, the connection drop caused it to hang, but now I'm not so 
sure. Were there any files I should maybe have cleared before trying to 
run it again?


I'm using the docker image to run the command (using 
osrm/osrm-backend:latest): time docker run -t -v /opt/osrm/data:/data 
osrm/osrm-backend osrm-contract /data/1505492056/planet-latest.osrm


Kind regards,
Kieran Caplice


___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk