Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Great, thanks again for your help.
Have a good week-end,
Patrick

2018-05-11 14:46 GMT-04:00 Sasha Khapyorsky :

> Technically it is possible (for example
> https://stackoverflow.com/questions/145270/calling-c-c-from-python),
> but I'm pretty sure that you will need to create sort of c++ envelop,
> shared lib, etc.. It would be easier just to do what you need in c++
> instead of python.
>
> Sasha
>
> On Fri, May 11, 2018 at 9:37 PM, Patrick Agin 
> wrote:
> > thanks Sasha. And do you know if it's possible to do calls to libosrm
> > functions directly from Python?
> >
> > 2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky :
> >>
> >> Hi again, Patrick,
> >>
> >> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin 
> >> wrote:
> >> >
> >> > And are you aware of some python code that would do the calls to
> >> > osrm-routed
> >> > in parallel threads?
> >>
> >> There are lot of examples of how to make python things in parallel
> >> threads. For example:
> >> https://stackoverflow.com/questions/2846653/how-to-use-
> threading-in-python
> >>
> >> Sasha
> >>
> >> > Regards,
> >> > Patrick
> >> >
> >> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson :
> >> >>
> >> >> Patrick,
> >> >>
> >> >>   There are about a million possible paths you could take here, a lot
> >> >> of
> >> >> it will depend on what skills you have available.  Off the top of my
> >> >> head:
> >> >>
> >> >> 1) Speed things up by avoiding HTTP overhead and calling the
> >> >> libosrm.a
> >> >> functions directly instead of hitting `osrm-routed` over HTTP
> >> >> 2) Modify the OSRM C++ source code and strip out the parts of the
> >> >> map-matching response you don't need
> >> >> 3) Simplify your trace geometries to speed up map-matching
> >> >> 4) Break your trace list into sets and run these on multiple
> >> >> machines
> >> >> in parallel (make copies of the OSRM data onto multiple machines)
> >> >> 5) Just wait 10 hours, and get a good nights sleep
> >> >>
> >> >>   libosrm.a is thread-safe, so if you're calling functions from
> threads
> >> >> you can do many at once.
> >> >>
> >> >>   osrm-routed is multi-threaded, so you can run many queries in
> >> >> parallel -
> >> >> how many will depend on how many CPUs your machine has.  Profiling
> >> >> multi-threaded server performance is kind of beyond the scope of OSRM
> >> >> itself, there is lots of literature on it.
> >> >>
> >> >> daniel
> >> >>
> >> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin <
> agin.patr...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Sorry for the newbie question but what's the difference between
> >> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
> >> >>> parallel
> >> >>> usage? And do you have an example of code that does the calls in
> >> >>> parallel
> >> >>> threads? Thanks Sasha for your help.
> >> >>> Patrick
> >> >>>
> >> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
> >> 
> >>  Hi Patrick,
> >> 
> >>  If you are using libosrm (which reported to be thread safe:
> >>  https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
> >>  just
> >>  split your list and run its parts in different parallel threads.
> >> 
> >>  Sasha
> >> 
> >>  On Fri, May 11, 2018 at 8:14 PM, Patrick Agin
> >>  
> >>  wrote:
> >>  > Hi,
> >>  > I want to call the map matching service for nearly a million of
> >>  > routes. I
> >>  > calculated that the task could take 10 hours on my laptop to
> >>  > complete.
> >>  > I was
> >>  > wondering if there's someone who faced the same problem and who
> >>  > addressed it
> >>  > in parallel, maybe with Spark? Another tool or way?
> >>  > Any hint is greatly appreciated!
> >>  > Patrick
> >>  >
> >>  > ___
> >>  > OSRM-talk mailing list
> >>  > OSRM-talk@openstreetmap.org
> >>  > https://lists.openstreetmap.org/listinfo/osrm-talk
> >>  >
> >> 
> >>  ___
> >>  OSRM-talk mailing list
> >>  OSRM-talk@openstreetmap.org
> >>  https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>>
> >> >>>
> >> >>> ___
> >> >>> OSRM-talk mailing list
> >> >>> OSRM-talk@openstreetmap.org
> >> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>
> >> >>
> >> >> ___
> >> >> OSRM-talk mailing list
> >> >> OSRM-talk@openstreetmap.org
> >> >> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>
> >> >
> >> >
> >> > ___
> >> > OSRM-talk mailing list
> >> > OSRM-talk@openstreetmap.org
> >> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >
> >>
> >> 

Re: [OSRM-talk] map matching service for a huge number of requests

[Daniel Patterson]

Some folks have written Python wrappers around libosrm, like here:

  https://github.com/ustroetz/python-osrm

but I've never tried them, so I don't know how up-to-date they are, or how
easy they will be to get working.

If you're comfortable in NodeJS, then OSRM supports Node bindings as a
first-class citizen.

daniel

On Fri, May 11, 2018 at 11:46 AM, Sasha Khapyorsky 
wrote:

> Technically it is possible (for example
> https://stackoverflow.com/questions/145270/calling-c-c-from-python),
> but I'm pretty sure that you will need to create sort of c++ envelop,
> shared lib, etc.. It would be easier just to do what you need in c++
> instead of python.
>
> Sasha
>
> On Fri, May 11, 2018 at 9:37 PM, Patrick Agin 
> wrote:
> > thanks Sasha. And do you know if it's possible to do calls to libosrm
> > functions directly from Python?
> >
> > 2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky :
> >>
> >> Hi again, Patrick,
> >>
> >> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin 
> >> wrote:
> >> >
> >> > And are you aware of some python code that would do the calls to
> >> > osrm-routed
> >> > in parallel threads?
> >>
> >> There are lot of examples of how to make python things in parallel
> >> threads. For example:
> >> https://stackoverflow.com/questions/2846653/how-to-use-
> threading-in-python
> >>
> >> Sasha
> >>
> >> > Regards,
> >> > Patrick
> >> >
> >> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson :
> >> >>
> >> >> Patrick,
> >> >>
> >> >>   There are about a million possible paths you could take here, a lot
> >> >> of
> >> >> it will depend on what skills you have available.  Off the top of my
> >> >> head:
> >> >>
> >> >> 1) Speed things up by avoiding HTTP overhead and calling the
> >> >> libosrm.a
> >> >> functions directly instead of hitting `osrm-routed` over HTTP
> >> >> 2) Modify the OSRM C++ source code and strip out the parts of the
> >> >> map-matching response you don't need
> >> >> 3) Simplify your trace geometries to speed up map-matching
> >> >> 4) Break your trace list into sets and run these on multiple
> >> >> machines
> >> >> in parallel (make copies of the OSRM data onto multiple machines)
> >> >> 5) Just wait 10 hours, and get a good nights sleep
> >> >>
> >> >>   libosrm.a is thread-safe, so if you're calling functions from
> threads
> >> >> you can do many at once.
> >> >>
> >> >>   osrm-routed is multi-threaded, so you can run many queries in
> >> >> parallel -
> >> >> how many will depend on how many CPUs your machine has.  Profiling
> >> >> multi-threaded server performance is kind of beyond the scope of OSRM
> >> >> itself, there is lots of literature on it.
> >> >>
> >> >> daniel
> >> >>
> >> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin <
> agin.patr...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Sorry for the newbie question but what's the difference between
> >> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
> >> >>> parallel
> >> >>> usage? And do you have an example of code that does the calls in
> >> >>> parallel
> >> >>> threads? Thanks Sasha for your help.
> >> >>> Patrick
> >> >>>
> >> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
> >> 
> >>  Hi Patrick,
> >> 
> >>  If you are using libosrm (which reported to be thread safe:
> >>  https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
> >>  just
> >>  split your list and run its parts in different parallel threads.
> >> 
> >>  Sasha
> >> 
> >>  On Fri, May 11, 2018 at 8:14 PM, Patrick Agin
> >>  
> >>  wrote:
> >>  > Hi,
> >>  > I want to call the map matching service for nearly a million of
> >>  > routes. I
> >>  > calculated that the task could take 10 hours on my laptop to
> >>  > complete.
> >>  > I was
> >>  > wondering if there's someone who faced the same problem and who
> >>  > addressed it
> >>  > in parallel, maybe with Spark? Another tool or way?
> >>  > Any hint is greatly appreciated!
> >>  > Patrick
> >>  >
> >>  > ___
> >>  > OSRM-talk mailing list
> >>  > OSRM-talk@openstreetmap.org
> >>  > https://lists.openstreetmap.org/listinfo/osrm-talk
> >>  >
> >> 
> >>  ___
> >>  OSRM-talk mailing list
> >>  OSRM-talk@openstreetmap.org
> >>  https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>>
> >> >>>
> >> >>> ___
> >> >>> OSRM-talk mailing list
> >> >>> OSRM-talk@openstreetmap.org
> >> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>
> >> >>
> >> >> ___
> >> >> OSRM-talk mailing list
> >> >> OSRM-talk@openstreetmap.org
> >> >> 

Re: [OSRM-talk] map matching service for a huge number of requests

[Sasha Khapyorsky]

Technically it is possible (for example
https://stackoverflow.com/questions/145270/calling-c-c-from-python),
but I'm pretty sure that you will need to create sort of c++ envelop,
shared lib, etc.. It would be easier just to do what you need in c++
instead of python.

Sasha

On Fri, May 11, 2018 at 9:37 PM, Patrick Agin  wrote:
> thanks Sasha. And do you know if it's possible to do calls to libosrm
> functions directly from Python?
>
> 2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky :
>>
>> Hi again, Patrick,
>>
>> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin 
>> wrote:
>> >
>> > And are you aware of some python code that would do the calls to
>> > osrm-routed
>> > in parallel threads?
>>
>> There are lot of examples of how to make python things in parallel
>> threads. For example:
>> https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python
>>
>> Sasha
>>
>> > Regards,
>> > Patrick
>> >
>> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson :
>> >>
>> >> Patrick,
>> >>
>> >>   There are about a million possible paths you could take here, a lot
>> >> of
>> >> it will depend on what skills you have available.  Off the top of my
>> >> head:
>> >>
>> >> 1) Speed things up by avoiding HTTP overhead and calling the
>> >> libosrm.a
>> >> functions directly instead of hitting `osrm-routed` over HTTP
>> >> 2) Modify the OSRM C++ source code and strip out the parts of the
>> >> map-matching response you don't need
>> >> 3) Simplify your trace geometries to speed up map-matching
>> >> 4) Break your trace list into sets and run these on multiple
>> >> machines
>> >> in parallel (make copies of the OSRM data onto multiple machines)
>> >> 5) Just wait 10 hours, and get a good nights sleep
>> >>
>> >>   libosrm.a is thread-safe, so if you're calling functions from threads
>> >> you can do many at once.
>> >>
>> >>   osrm-routed is multi-threaded, so you can run many queries in
>> >> parallel -
>> >> how many will depend on how many CPUs your machine has.  Profiling
>> >> multi-threaded server performance is kind of beyond the scope of OSRM
>> >> itself, there is lots of literature on it.
>> >>
>> >> daniel
>> >>
>> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin 
>> >> wrote:
>> >>>
>> >>> Sorry for the newbie question but what's the difference between
>> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
>> >>> parallel
>> >>> usage? And do you have an example of code that does the calls in
>> >>> parallel
>> >>> threads? Thanks Sasha for your help.
>> >>> Patrick
>> >>>
>> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
>> 
>>  Hi Patrick,
>> 
>>  If you are using libosrm (which reported to be thread safe:
>>  https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
>>  just
>>  split your list and run its parts in different parallel threads.
>> 
>>  Sasha
>> 
>>  On Fri, May 11, 2018 at 8:14 PM, Patrick Agin
>>  
>>  wrote:
>>  > Hi,
>>  > I want to call the map matching service for nearly a million of
>>  > routes. I
>>  > calculated that the task could take 10 hours on my laptop to
>>  > complete.
>>  > I was
>>  > wondering if there's someone who faced the same problem and who
>>  > addressed it
>>  > in parallel, maybe with Spark? Another tool or way?
>>  > Any hint is greatly appreciated!
>>  > Patrick
>>  >
>>  > ___
>>  > OSRM-talk mailing list
>>  > OSRM-talk@openstreetmap.org
>>  > https://lists.openstreetmap.org/listinfo/osrm-talk
>>  >
>> 
>>  ___
>>  OSRM-talk mailing list
>>  OSRM-talk@openstreetmap.org
>>  https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>>
>> >>>
>> >>>
>> >>> ___
>> >>> OSRM-talk mailing list
>> >>> OSRM-talk@openstreetmap.org
>> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>>
>> >>
>> >>
>> >> ___
>> >> OSRM-talk mailing list
>> >> OSRM-talk@openstreetmap.org
>> >> https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>
>> >
>> >
>> > ___
>> > OSRM-talk mailing list
>> > OSRM-talk@openstreetmap.org
>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>> >
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org

Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

thanks Sasha. And do you know if it's possible to do calls to libosrm
functions directly from Python?

2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky :

> Hi again, Patrick,
>
> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin 
> wrote:
> >
> > And are you aware of some python code that would do the calls to
> osrm-routed
> > in parallel threads?
>
> There are lot of examples of how to make python things in parallel
> threads. For example:
> https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python
>
> Sasha
>
> > Regards,
> > Patrick
> >
> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson :
> >>
> >> Patrick,
> >>
> >>   There are about a million possible paths you could take here, a lot of
> >> it will depend on what skills you have available.  Off the top of my
> head:
> >>
> >> 1) Speed things up by avoiding HTTP overhead and calling the
> libosrm.a
> >> functions directly instead of hitting `osrm-routed` over HTTP
> >> 2) Modify the OSRM C++ source code and strip out the parts of the
> >> map-matching response you don't need
> >> 3) Simplify your trace geometries to speed up map-matching
> >> 4) Break your trace list into sets and run these on multiple
> machines
> >> in parallel (make copies of the OSRM data onto multiple machines)
> >> 5) Just wait 10 hours, and get a good nights sleep
> >>
> >>   libosrm.a is thread-safe, so if you're calling functions from threads
> >> you can do many at once.
> >>
> >>   osrm-routed is multi-threaded, so you can run many queries in
> parallel -
> >> how many will depend on how many CPUs your machine has.  Profiling
> >> multi-threaded server performance is kind of beyond the scope of OSRM
> >> itself, there is lots of literature on it.
> >>
> >> daniel
> >>
> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin 
> >> wrote:
> >>>
> >>> Sorry for the newbie question but what's the difference between
> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
> parallel
> >>> usage? And do you have an example of code that does the calls in
> parallel
> >>> threads? Thanks Sasha for your help.
> >>> Patrick
> >>>
> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
> 
>  Hi Patrick,
> 
>  If you are using libosrm (which reported to be thread safe:
>  https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
> just
>  split your list and run its parts in different parallel threads.
> 
>  Sasha
> 
>  On Fri, May 11, 2018 at 8:14 PM, Patrick Agin  >
>  wrote:
>  > Hi,
>  > I want to call the map matching service for nearly a million of
>  > routes. I
>  > calculated that the task could take 10 hours on my laptop to
> complete.
>  > I was
>  > wondering if there's someone who faced the same problem and who
>  > addressed it
>  > in parallel, maybe with Spark? Another tool or way?
>  > Any hint is greatly appreciated!
>  > Patrick
>  >
>  > ___
>  > OSRM-talk mailing list
>  > OSRM-talk@openstreetmap.org
>  > https://lists.openstreetmap.org/listinfo/osrm-talk
>  >
> 
>  ___
>  OSRM-talk mailing list
>  OSRM-talk@openstreetmap.org
>  https://lists.openstreetmap.org/listinfo/osrm-talk
> >>>
> >>>
> >>>
> >>> ___
> >>> OSRM-talk mailing list
> >>> OSRM-talk@openstreetmap.org
> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
> >>>
> >>
> >>
> >> ___
> >> OSRM-talk mailing list
> >> OSRM-talk@openstreetmap.org
> >> https://lists.openstreetmap.org/listinfo/osrm-talk
> >>
> >
> >
> > ___
> > OSRM-talk mailing list
> > OSRM-talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Sasha Khapyorsky]

Hi again, Patrick,

On Fri, May 11, 2018 at 9:22 PM, Patrick Agin  wrote:
>
> And are you aware of some python code that would do the calls to osrm-routed
> in parallel threads?

There are lot of examples of how to make python things in parallel
threads. For example:
https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python

Sasha

> Regards,
> Patrick
>
> 2018-05-11 14:07 GMT-04:00 Daniel Patterson :
>>
>> Patrick,
>>
>>   There are about a million possible paths you could take here, a lot of
>> it will depend on what skills you have available.  Off the top of my head:
>>
>> 1) Speed things up by avoiding HTTP overhead and calling the libosrm.a
>> functions directly instead of hitting `osrm-routed` over HTTP
>> 2) Modify the OSRM C++ source code and strip out the parts of the
>> map-matching response you don't need
>> 3) Simplify your trace geometries to speed up map-matching
>> 4) Break your trace list into sets and run these on multiple machines
>> in parallel (make copies of the OSRM data onto multiple machines)
>> 5) Just wait 10 hours, and get a good nights sleep
>>
>>   libosrm.a is thread-safe, so if you're calling functions from threads
>> you can do many at once.
>>
>>   osrm-routed is multi-threaded, so you can run many queries in parallel -
>> how many will depend on how many CPUs your machine has.  Profiling
>> multi-threaded server performance is kind of beyond the scope of OSRM
>> itself, there is lots of literature on it.
>>
>> daniel
>>
>> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin 
>> wrote:
>>>
>>> Sorry for the newbie question but what's the difference between
>>> osrm-routed and libosrm? Is it mandatory to use the latter for a parallel
>>> usage? And do you have an example of code that does the calls in parallel
>>> threads? Thanks Sasha for your help.
>>> Patrick
>>>
>>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :

 Hi Patrick,

 If you are using libosrm (which reported to be thread safe:
 https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
 split your list and run its parts in different parallel threads.

 Sasha

 On Fri, May 11, 2018 at 8:14 PM, Patrick Agin 
 wrote:
 > Hi,
 > I want to call the map matching service for nearly a million of
 > routes. I
 > calculated that the task could take 10 hours on my laptop to complete.
 > I was
 > wondering if there's someone who faced the same problem and who
 > addressed it
 > in parallel, maybe with Spark? Another tool or way?
 > Any hint is greatly appreciated!
 > Patrick
 >
 > ___
 > OSRM-talk mailing list
 > OSRM-talk@openstreetmap.org
 > https://lists.openstreetmap.org/listinfo/osrm-talk
 >

 ___
 OSRM-talk mailing list
 OSRM-talk@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osrm-talk
>>>
>>>
>>>
>>> ___
>>> OSRM-talk mailing list
>>> OSRM-talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>>
>>
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] Any existing tool to digest OSRM navi data?

[Jose Florido]

Maybe this will help you: http://www.liedman.net/leaflet-routing-machine/

On Fri, May 11, 2018 at 8:23 PM, Xavier Prudent 
wrote:

> Dear all,
>
> Under the option steps=true, osrm returns the detailed list of turn by
> turn steps along the itinerary.
>
> Does any tool already exist that can understand and display such
> information?
>
> Regards,
>
> Xavier
>
> --
>
> *Xavier Prudent *
>
> *Data Scientist  - Data Mining - Machine Learning*
>
> Web:* www.xavierprudent.com *
> Tel (Québec)  : (514) 668 76 46
> Skype : xavierprudent
>
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

[OSRM-talk] Any existing tool to digest OSRM navi data?

[Xavier Prudent]

Dear all,

Under the option steps=true, osrm returns the detailed list of turn by turn
steps along the itinerary.

Does any tool already exist that can understand and display such
information?

Regards,

Xavier

-- 

*Xavier Prudent *

*Data Scientist  - Data Mining - Machine Learning*

Web:* www.xavierprudent.com *
Tel (Québec)  : (514) 668 76 46
Skype : xavierprudent
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Thanks Daniel. My first idea was to break my list into sets and running the
job on different machines as you said in 4).
But out of curiosity, can I make calls to libosrm.a functions directly from
Python?
And are you aware of some python code that would do the calls to
osrm-routed in parallel threads?
Regards,
Patrick

2018-05-11 14:07 GMT-04:00 Daniel Patterson :

> Patrick,
>
>   There are about a million possible paths you could take here, a lot of
> it will depend on what skills you have available.  Off the top of my head:
>
> 1) Speed things up by avoiding HTTP overhead and calling the libosrm.a
> functions directly instead of hitting `osrm-routed` over HTTP
> 2) Modify the OSRM C++ source code and strip out the parts of the
> map-matching response you don't need
> 3) Simplify your trace geometries to speed up map-matching
> 4) Break your trace list into sets and run these on multiple machines
> in parallel (make copies of the OSRM data onto multiple machines)
> 5) Just wait 10 hours, and get a good nights sleep
>
>   libosrm.a is thread-safe, so if you're calling functions from threads
> you can do many at once.
>
>   osrm-routed is multi-threaded, so you can run many queries in parallel -
> how many will depend on how many CPUs your machine has.  Profiling
> multi-threaded server performance is kind of beyond the scope of OSRM
> itself, there is lots of literature on it.
>
> daniel
>
> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin 
> wrote:
>
>> Sorry for the newbie question but what's the difference between
>> osrm-routed and libosrm? Is it mandatory to use the latter for a parallel
>> usage? And do you have an example of code that does the calls in parallel
>> threads? Thanks Sasha for your help.
>> Patrick
>>
>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
>>
>>> Hi Patrick,
>>>
>>> If you are using libosrm (which reported to be thread safe:
>>> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
>>> split your list and run its parts in different parallel threads.
>>>
>>> Sasha
>>>
>>> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin 
>>> wrote:
>>> > Hi,
>>> > I want to call the map matching service for nearly a million of
>>> routes. I
>>> > calculated that the task could take 10 hours on my laptop to complete.
>>> I was
>>> > wondering if there's someone who faced the same problem and who
>>> addressed it
>>> > in parallel, maybe with Spark? Another tool or way?
>>> > Any hint is greatly appreciated!
>>> > Patrick
>>> >
>>> > ___
>>> > OSRM-talk mailing list
>>> > OSRM-talk@openstreetmap.org
>>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>>> >
>>>
>>> ___
>>> OSRM-talk mailing list
>>> OSRM-talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>>
>>
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Sasha Khapyorsky]

Hi again,

On Fri, May 11, 2018 at 9:01 PM, Patrick Agin  wrote:
> Ok I found some explanations about libosrm here:
> https://github.com/Project-OSRM/osrm-backend/blob/master/docs/libosrm.md
> My second questions remains though: is libosrm mandatory for parallel use?

It is not mandatory. Just faster.

> or could I use osrm-routed too (even if it is slower)?
> All my code is written is Python so it could be great if I can integrate
> python code to do the calls in parallel.

The same you can do with Python run osrm-routed request in parallel
threads (or even parallel processes).

Sasha

> Thanks again
> P.
>
> 2018-05-11 13:57 GMT-04:00 Patrick Agin :
>>
>> Sorry for the newbie question but what's the difference between
>> osrm-routed and libosrm? Is it mandatory to use the latter for a parallel
>> usage? And do you have an example of code that does the calls in parallel
>> threads? Thanks Sasha for your help.
>> Patrick
>>
>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
>>>
>>> Hi Patrick,
>>>
>>> If you are using libosrm (which reported to be thread safe:
>>> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
>>> split your list and run its parts in different parallel threads.
>>>
>>> Sasha
>>>
>>> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin 
>>> wrote:
>>> > Hi,
>>> > I want to call the map matching service for nearly a million of routes.
>>> > I
>>> > calculated that the task could take 10 hours on my laptop to complete.
>>> > I was
>>> > wondering if there's someone who faced the same problem and who
>>> > addressed it
>>> > in parallel, maybe with Spark? Another tool or way?
>>> > Any hint is greatly appreciated!
>>> > Patrick
>>> >
>>> > ___
>>> > OSRM-talk mailing list
>>> > OSRM-talk@openstreetmap.org
>>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>>> >
>>>
>>> ___
>>> OSRM-talk mailing list
>>> OSRM-talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>>
>
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Ok I found some explanations about libosrm here:
https://github.com/Project-OSRM/osrm-backend/blob/master/docs/libosrm.md
My second questions remains though: is libosrm mandatory for parallel use?
or could I use osrm-routed too (even if it is slower)?
All my code is written is Python so it could be great if I can integrate
python code to do the calls in parallel.
Thanks again
P.

2018-05-11 13:57 GMT-04:00 Patrick Agin :

> Sorry for the newbie question but what's the difference between
> osrm-routed and libosrm? Is it mandatory to use the latter for a parallel
> usage? And do you have an example of code that does the calls in parallel
> threads? Thanks Sasha for your help.
> Patrick
>
> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :
>
>> Hi Patrick,
>>
>> If you are using libosrm (which reported to be thread safe:
>> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
>> split your list and run its parts in different parallel threads.
>>
>> Sasha
>>
>> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin 
>> wrote:
>> > Hi,
>> > I want to call the map matching service for nearly a million of routes.
>> I
>> > calculated that the task could take 10 hours on my laptop to complete.
>> I was
>> > wondering if there's someone who faced the same problem and who
>> addressed it
>> > in parallel, maybe with Spark? Another tool or way?
>> > Any hint is greatly appreciated!
>> > Patrick
>> >
>> > ___
>> > OSRM-talk mailing list
>> > OSRM-talk@openstreetmap.org
>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>> >
>>
>> ___
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Sorry for the newbie question but what's the difference between osrm-routed
and libosrm? Is it mandatory to use the latter for a parallel usage? And do
you have an example of code that does the calls in parallel threads? Thanks
Sasha for your help.
Patrick

2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky :

> Hi Patrick,
>
> If you are using libosrm (which reported to be thread safe:
> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
> split your list and run its parts in different parallel threads.
>
> Sasha
>
> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin 
> wrote:
> > Hi,
> > I want to call the map matching service for nearly a million of routes. I
> > calculated that the task could take 10 hours on my laptop to complete. I
> was
> > wondering if there's someone who faced the same problem and who
> addressed it
> > in parallel, maybe with Spark? Another tool or way?
> > Any hint is greatly appreciated!
> > Patrick
> >
> > ___
> > OSRM-talk mailing list
> > OSRM-talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Sasha Khapyorsky]

Hi Patrick,

If you are using libosrm (which reported to be thread safe:
https://github.com/Project-OSRM/osrm-backend/issues/4966) you can just
split your list and run its parts in different parallel threads.

Sasha

On Fri, May 11, 2018 at 8:14 PM, Patrick Agin  wrote:
> Hi,
> I want to call the map matching service for nearly a million of routes. I
> calculated that the task could take 10 hours on my laptop to complete. I was
> wondering if there's someone who faced the same problem and who addressed it
> in parallel, maybe with Spark? Another tool or way?
> Any hint is greatly appreciated!
> Patrick
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>

___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

Re: [OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Very interesting, thanks Bryan. Unfortunately, the requirement about Stata
is probably a downer for us.

2018-05-11 13:30 GMT-04:00 Sayer, Bryan :

> We use the Stata implementation OSRMTIME from Stephan Huber at Regensburg
> University  which can split the data across threads (we have 2 CPUS with
> four cores each, and two threads per core). So 16 threads on a 3.5 Ghz Xeon
> and 64 GB of RAM, and if the network traffic is low (like on holidays) I
> can get about 1.2 million routes per hour in the United States.
>
>
> Of course, this requires having Stata, but we find it is a pretty good way
> to do batch routing. I recently did about 460 million routes. But it only
> returns distance and time, not the actual route, if that matters.
>
>
> http://www.uni-regensburg.de/wirtschaftswissenschaften/vwl-
> moeller/medien/huber/osrm_paper_online.pdf
> osrmtime: Calculate Travel Time and Distance with ...
> 
> www.uni-regensburg.de
> The Stata Journal (2016) 16, Number , pp. 1{8 osrmtime: Calculate Travel
> Time and Distance with OpenStreetMap Data Using the Open Source Routing
> Machine (OSRM)
>
>
> --
> *From:* Patrick Agin 
> *Sent:* Friday, May 11, 2018 1:14:31 PM
> *To:* Mailing list to discuss Project OSRM
> *Subject:* [OSRM-talk] map matching service for a huge number of requests
>
> Hi,
> I want to call the map matching service for nearly a million of routes. I
> calculated that the task could take 10 hours on my laptop to complete. I
> was wondering if there's someone who faced the same problem and who
> addressed it in parallel, maybe with Spark? Another tool or way?
> Any hint is greatly appreciated!
> Patrick
>
> WARNING This information may be confidential. It is intended only
> for the addressee(s) identified above. If you are not the addressee(s), or
> an employee or agent of the addressee(s), please note that any
> dissemination, distribution, or copying of this communication is strictly
> prohibited. If you have received this information in error, please destroy
> the information and notify the sender of the error. Thank you.
>
> ___
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk

[OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Hi,
I want to call the map matching service for nearly a million of routes. I
calculated that the task could take 10 hours on my laptop to complete. I
was wondering if there's someone who faced the same problem and who
addressed it in parallel, maybe with Spark? Another tool or way?
Any hint is greatly appreciated!
Patrick
___
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk