Richard, For processing larger data sets quickly, consider using pay-by-the-hour computing resources, like Amazon EC2. Using a high-memory EC2 machine should be able to get EU processed in just a couple of hours.
As long as the architecture is the same as the machine you will run `osrm-routed` on, you should be able to copy the processed data files onto your target machine. Actual routing requires a lot less memory than processing. daniel > On Oct 15, 2015, at 6:43 AM, Richard Marsden <[email protected]> wrote: > > Thanks - I'll try to work through the profiles - hopefully they'll > make some sense as I work through them. > > osrm-prepare also took two weeks to crash.Perhaps it did have a sudden > spike in memory usage before crashing - it was overnight, so I don't > know for sure. Anyway I'm not in a rush to try another two week run. > > Cheers, > Richard > > On Wed, Oct 14, 2015 at 4:20 AM, Daniel Hofmann <[email protected]> wrote: >> If osrm-prepare crashed, just add more swap. Also make sure to read this >> ticket: >>> https://github.com/Project-OSRM/osrm-backend/issues/1698 >> >> I don't think we have documentation for the profiles. I know they are >> somewhat ugly and huge. I would recommend taking a look at the profile's >> general structure first, so that you roughly understand e.g. that there are >> callbacks for OSM nodes and ways, and what they do. After that I would >> simply trace the globals through the script. That is e.g. search for all >> places where a certain whitelist is used and build a mental model based on >> that. Pen and paper may help :) >> >>> https://github.com/Project-OSRM/osrm-backend/blob/develop/profiles/car.lua >> >> You can find the C++ side of things in the extractor directory: >>> https://github.com/Project-OSRM/osrm-backend/tree/develop/extractor >> >> For example, the scripting environment uses luabind to make a few functions >> visible, e.g. the get_value_by_key function, that then can be used in the >> profiles. The other way, that is using lua functions in C++ is mostly done >> in the extractor implementation: I would recommend just searching for the >> symbols you want to know more about: >> >>> >>> https://github.com/Project-OSRM/osrm-backend/search?utf8=%E2%9C%93&q=node_function&type=Code >> >> Cheers, >> Daniel >> >> On Mon, Oct 12, 2015 at 4:00 AM, Richard Marsden <[email protected]> wrote: >>> >>> It is possible I've just managed to demonstrate this to myself. >>> osrm-prepare crashed out on the Europe-wide dataset at the 90+% >>> complete level with 30GB swap (16GB real). For much of the run it >>> didn't go above about 15GB swap. Very slow of course - but that was to >>> be expected. >>> >>> This continues to be a side / evaluation project, but I'm continuing >>> to investigate. Looks like the PC in question should be able to handle >>> most countries, but will balk at most continents, and the very largest >>> countries (e.g. the US). High-RAM hardware is relatively pricey, but >>> it is a possibility in the new year. >>> >>> Looking at the individual pieces, next I'm going to look at the >>> profile.lua file. I've never used LUA before, but that is easy enough >>> to remedy with online tutorials and the various books that are >>> available. But what about the script that is used by OSRM? Is there >>> any documentation as to the variables and/or functions that are >>> expected? And the tags used in OSM that the script is expected to >>> map? >>> >>> >>> Cheers, >>> >>> Richard Marsden >>> >>> On Thu, Sep 17, 2015 at 4:37 PM, Patrick Niklaus >>> <[email protected]> wrote: >>>> W.r.t. the pre-preprocessing you are correct. >>>> >>>>> What is that extra power used for? >>>> >>>> Including all sorts of external data sources. Also the logic in the >>>> lua profiles is not just replaceable by simple key-value pairs, OSM >>>> requires you to handle a lot of special cases. >>>> >>>>> Presumably I could do the same for world preparation & routing? Have, >>>>> perhaps a 100GB+ swap file, ideally on an SSD. >>>> >>>> This will fall apart when you have some actual load pressure on the >>>> system. We need random access to memory, which will create a lot of >>>> page faults (== slow). Even an SSD is not even close to memory speed. >>>> >>>> You have two options: >>>> - split the datasets >>>> - get a bigger server >>>> >>>> Cheers, >>>> Patrick >>>> >>>> >>>> On Thu, Sep 17, 2015 at 10:06 PM, Richard Marsden <[email protected]> >>>> wrote: >>>>> I've been evaluating OSRM, using it primarily as a library from C++. >>>>> >>>>> I believe I've determined the answer to most of the questions, but I'm >>>>> also looking for confirmation. >>>>> (I understand the reason for these constraints - the trade-off of >>>>> speed vs flexibility) >>>>> >>>>> First, road speeds are set with 'profile.lua' at the osrm-extract >>>>> stage. This filters out unnecessary roads (eg. foot paths for car >>>>> routing), but also applies the road speeds. >>>>> If I wish to change the speed profile, I need to regenerate the road >>>>> network with osrm-extract and osrm-routed. >>>>> Correct? >>>>> >>>>> If I wanted different speeds for the final distance/time calculations, >>>>> I could use the returned route, and apply my own speed table according >>>>> to the road type of each road segment. This would not, of course, >>>>> change the route geometry is calculated. >>>>> >>>>> If I want a shortest route (distance optimized) instead of a quickest >>>>> route (time optimized), I need to set all the road speeds to the same >>>>> speed and regenerate the network. I.e. osrm does not directly support >>>>> the concept of a "shortest route". >>>>> >>>>> The profile is provided with a LUA file. I had to look this one up :-) >>>>> Looks a useful scripting language, but why is this profile a script >>>>> file, and not a simple configuration file of constants (eg. key-value >>>>> pairs)? >>>>> Seems like an unnecessary complexity - I'd like to understand the >>>>> perceived advantages. What is that extra power used for? >>>>> >>>>> Finally, the memory usage... I saw a reference to the server requiring >>>>> 40GB of memory for pan-European routing. Presumably that could be >>>>> offset with a large swap file(?) >>>>> A large swap file has worked well when I was testing the US-South >>>>> region on an 8GB machine. >>>>> Presumably I could do the same for world preparation & routing? Have, >>>>> perhaps a 100GB+ swap file, ideally on an SSD. >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Richard Marsden >>>>> >>>>> _______________________________________________ >>>>> OSRM-talk mailing list >>>>> [email protected] >>>>> https://lists.openstreetmap.org/listinfo/osrm-talk >>>> >>>> _______________________________________________ >>>> OSRM-talk mailing list >>>> [email protected] >>>> https://lists.openstreetmap.org/listinfo/osrm-talk >>> >>> _______________________________________________ >>> OSRM-talk mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/osrm-talk >> >> >> >> _______________________________________________ >> OSRM-talk mailing list >> [email protected] >> https://lists.openstreetmap.org/listinfo/osrm-talk >> > > _______________________________________________ > OSRM-talk mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/osrm-talk _______________________________________________ OSRM-talk mailing list [email protected] https://lists.openstreetmap.org/listinfo/osrm-talk
