Hi Paul, On 18 April 2013 17:36, Paul Norman <[email protected]> wrote:
> I'm considering making use of streaming replication to feed change data to > a > process, but I'm not quite sure how to script it to handle interruptions > like power outages and such correctly without getting in advance of itself. > Does anyone have any wrapper scripts that would help with this? > > I'd rather not reinvent the wheel so I can focus on the logic of > interpreting the osc data and putting it into my database. > > Alternately, I could re-implement a streaming replication reader in python, > which is what I intend to be writing my osc to DB code in. > It'd be great to see the replication streaming getting some usage. It doesn't seem to be used at the moment. In fact it's been down for a couple of weeks and it's trying to catch up again now. I assume you have a custom database schema that you're writing to. In other words, I assume you can't use existing Osmosis tasks to write to your database. There are a couple of example Osmosis command lines on the following wiki page in case you haven't already seen them. http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming If you're writing your own database writer, you'll probably have to use Osmosis to write to standard out, and then pipe data into standard input in your program. The problem you'll face with this approach is that Osmosis will assume that written to standard output has been successfully processed and will update its local state.txt accordingly. It has no way of knowing when the downstream process has finished processing the data and has committed it. If a crash occurs between Osmosis updating the state.txt file and your application committing changes to the database, those changes will be lost. I don't think it's possible to solve this using standard piping between processes, it would require some two way communication. It may be simpler to implement a new Osmosis plugin for your database, or to re-implement the client streaming functionality. Let me know if the above doesn't make sense :-) Brett
_______________________________________________ osmosis-dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/osmosis-dev
