PBF reading is definitely not IO bound. It is actually quite expensive to parse PBF, so, yes, it about everything you do multi-threaded reading is going to be a win.
Jochen On Do, Jun 04, 2015 at 11:04:48 +0200, Andrew Byrd wrote: > Can anyone provide anecdotes of use cases where multi-threaded PBF reading > significantly speeds up processing? Generally I would expect PBF reading to > be IO bound rather than processor bound, but I still need to make more > accurate measurements. > > Of course actually processing the OSM data once the PBF is decoded can be > quite CPU intensive, but that would imply buffering decoded data and > parallelizing geometric operations for example, not the reading. > > I’d appreciate any data points and example use cases you might have, as I’m > currently working on related tooling. > > Andrew Byrd > > > On 04 Jun 2015, at 05:57, Brett Henderson <br...@bretth.com> wrote: > > > > On 30 April 2015 at 03:27, Paul Norman <penor...@mac.com > > <mailto:penor...@mac.com>> wrote: > > On 4/29/2015 9:55 AM, Martijn van Exel wrote: > > If osmosis is the reference implementation, is there a reason why it > > doesn't seem to leverage this block structure to speed up reading? Or > > does it? > > Osmosis has the --read-pbf-fast task which allows multiple worker threads. > > > > That's right. I forget how the PBF structure works off the top of my head, > > but the file is already split into blocks. The main --read-pbf-fast thread > > simply grabs the outer protobuf blocks from file and then distributes them > > to worker threads who parse out the OSM entities from within the block. > > After extraction, the entities within each block are passed to the > > downstream task in original file order. I'm not sure I see the need to > > modify the PBF file format. > > > > _______________________________________________ > > osmosis-dev mailing list > > osmosis-dev@openstreetmap.org > > https://lists.openstreetmap.org/listinfo/osmosis-dev > > _______________________________________________ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-351-31778688 _______________________________________________ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
