Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-12 Thread Brett Henderson
Hi Igor, On Sat, Jun 4, 2011 at 6:04 AM, Igor Podolskiy < igor.podols...@vwi-stuttgart.de> wrote: > Hi Brett, hi @osmosis-dev, > > > You're describing a common problem with the Osmosis pipeline. Many >> scenarios would be improved if you could access the data stream twice >> rather than having

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-08 Thread Peter Körner
Dear Developers Am 08.06.2011 13:47, schrieb Frederik Ramm: For example, there's "osmconvert" (http://wiki.openstreetmap.org/wiki/Osmconvert), a tool that can read/write OSM, apply diffs, do bounding box, polygon, and tag filtering, and people published benchmarks in the forum where that program

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-08 Thread Frederik Ramm
Hi, Jochen Topf wrote: I have thoght about these things quite a lot, too. I agree with Igor that we are bumping into the limits of what the current Osmosis pipelining model can do. We seem to be reaching a point where people are creating specialist programs to do things that are traditionally

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-08 Thread Jochen Topf
On Fri, Jun 03, 2011 at 10:04:40PM +0200, Igor Podolskiy wrote: > [lots of good thinking] I have thoght about these things quite a lot, too. I agree with Igor that we are bumping into the limits of what the current Osmosis pipelining model can do. And it gets even more complicated: For lots of the

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-07 Thread Guillaume Rischard
Hi Igor, On 3 Jun 2011, at 22:04, Igor Podolskiy wrote: > What I think we need is a much more generic solution, like a second type of > communication channel between the tasks for flow control information. This was a flawed idea with FTP. Let's not invent this again. Cheers, Guillaume ___

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-03 Thread Igor Podolskiy
Hi Brett, hi @osmosis-dev, You're describing a common problem with the Osmosis pipeline. Many scenarios would be improved if you could access the data stream twice rather than having to buffer all data within the task itself. good to know I'm not alone ;) In the meantime I've been working qui

Re: [osmosis-dev] --used-node performance and a possible way to improve it

2011-06-03 Thread Brett Henderson
Hi Igor, You're describing a common problem with the Osmosis pipeline. Many scenarios would be improved if you could access the data stream twice rather than having to buffer all data within the task itself. Your solution of requiring two input streams is quite practical. I've often been tempte

[osmosis-dev] --used-node performance and a possible way to improve it

2011-05-11 Thread Igor Podolskiy
Hi Osmosis developers, recently I found myself filtering large extracts of data with fairly restrictive tag-based filters. Imagine a task like "get all administrative boundaries in the state of Baden-Württemberg, Germany". It was taking a long time, lots of CPU, and lots of I/O. And I asked m