Very interesting work! How do you handle new incoming requests. They have to start from a known point so I guess you have to do an SQL query for each of them? Or do you just read the existing .osc files from disk and stream them out? This could take a long time...
Jochen On Sat, Oct 13, 2012 at 03:43:32PM +1100, Brett Henderson wrote: > Date: Sat, 13 Oct 2012 15:43:32 +1100 > From: Brett Henderson <[email protected]> > To: osmosis-dev <[email protected]> > Subject: [osmosis-dev] Streaming Replication > > Hi All, > > For those of you who currently use the minute diffs to keep a local > database up to date, you may be interested to know that a new form of > replication has hit the street. > > The current replication system is based on a series of static replication > files that are placed on a web server for clients to download as described > here: > http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Using_the_replication_diffs > > It is a very simple mechanism and works well for the existing daily, hourly > and minutely replication feeds. Unfortunately it doesn't work well for > sub-minute replication because it becomes far too "chatty". On the server > side, the current feeds are generated from cron which also works well down > to one minute intervals, but the overhead of launching a new process and > connecting to the database for every replication interval also becomes too > inefficient for shorter intervals. > > To solve this, a new streaming replication mechanism has been developed. > Under the covers the same database queries are utilised, but the process > performing the queries runs continously and polls the database for changes > at a shorter interval. It is currently set to poll every 10 seconds, but > it can be reduced further if required. The network transport is also > continuous and holds a single HTTP connection open for the lifetime of > communication between the server and client. It is all implemented within > the latest version of Osmosis 0.41. If you wish to experiment with the > server-side tasks however, several bugs have been fixed in the latest > development version. Internally it uses the JBoss Netty framework which > means that it's all event-driven (ie. doesn't require a thread per client) > and should theoretically support a large number of concurrent clients. > > To quickly see this in action, point your browser at this URL and you > should see new replication "state" data become available approximately > every 10 seconds. > http://planet.openstreetmap.org/replication/streaming/replicationState/current/tail > > New Osmosis tasks have been developed to consume this data. For some basic > instructions to help you get started, refer to this link: > http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming > > If you don't wish to use Osmosis, some basic documentation on the wire > protocol is available here: > http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol > > This is very much experimental and bugs will undoubtedly be encountered to > please be wary about trusting it to update your database if you've just > spent two weeks importing a planet file. However, I'd love to see it get > some usage and would welcome any feedback. This is not intended for use in > updating a local planet file as the existing daily files are better suited > to that. For databases that can tolerate a minute delay, the existing > mechanism is very simple and has proven to be fairly reliable. But if you > really need current access to data, and can cope with the additional > complexity, this should be useful. The current 10 second delay is not a > lower limit, but is a good starting point for now. > > Cheers, > Brett > _______________________________________________ > osmosis-dev mailing list > [email protected] > http://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf [email protected] http://www.remote.org/jochen/ +49-721-388298 _______________________________________________ osmosis-dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/osmosis-dev
