Re: [OSM-dev] TRAPI status
On Sun, Mar 14, 2010 at 9:56 PM, John Smith deltafoxtrot...@gmail.comwrote: On 14 March 2010 14:58, Brett Henderson br...@bretth.com wrote: In the meantime, if you do need something more regular than minute updates, the existing Osmosis mechanism should be able to get down to around 5 second intervals. I'm not keen to publish files publicly with that type of I haven't looked into this but I'm assuming some kind of push to the tile DB from the DB server as it accepts new data. It'd be nice if that could be expanded so others can subscribe to such updates as well. Sorry John, I'm not sure what you mean. What tile DB are you referring to, and what does it have to do with replication? Unless you're referring to TRAPI, but if that's the case we went off topic long ago ;-) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On 16 March 2010 19:29, Brett Henderson br...@bretth.com wrote: Sorry John, I'm not sure what you mean. What tile DB are you referring to, and what does it have to do with replication? Unless you're referring to TRAPI, but if that's the case we went off topic long ago ;-) OSM runs at least 2 DBs, one for the data and one tiles, the tile information is a subset. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
Hi, John Smith wrote: OSM runs at least 2 DBs, one for the data and one tiles, the tile information is a subset. The tile server uses the same minutely diffs that everyone else can already use, so no special replication there. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Fri, Mar 12, 2010 at 11:24 PM, Lars Francke lars.fran...@gmail.comwrote: Most OSM systems tend to have a large number of disorganised and uncontrolled clients. Does this work well with the AMQP paradigm? In other words, does it take administrative overhead to register new subscriptions to a queue? What happens if large numbers of subscriptions are created then the clients disappear? Is AMQP targetted at a world where the clients are relatively controlled and small in number? It's important to minimise administration overhead where possible. I'll try to answer those in order. Subscriptions in AMQP are defined client side[1] so there is _no_ administrative overhead at all. Clients only need to know which exchange/queue they should bind to and that's just a string they need to know. Queues can be declared in different ways, one way is to declare it as autoDelete[2]: true if we are declaring an autodelete queue (server will delete it when no longer in use). Again no administrative overhead. It just disappears when no longer used so no messages are routed there. So depending on the use case there are multiple options. I have no huge real-world experience with AMQP but I believe it is targeted at both worlds: Small and controlled and large and uncontrolled. The structure they have found seems to work really well in any case. Since the installation of the RabbitMQ server I did not have to touch it again. The only thing one _could_ do is to add some kind of user management (currently everyone is allowed to do everything for example send fake messages[4]) to allow only certain users to publish to certain exchanges but that are two lines in the admin console. Okay, that all sounds promising. My main queueing experience is with MQ Series, or WebSphere MQ, or whatever it's called these days. It's a little more top heavy :-) Clients will experience outages whether that be due to a network problem, server reboot, or just not running a system 24x7. Presumably they need a way to catch up on missed events. There are a few options: 1. The server holds events for clients until they become available again, 2. The client catches up using an out of band mechanism (eg. downloads diffs directly), or 3. The client can request that the server begin sending data from a specific point. I think that only options 1 and 2 are possible using AMQP. 1 is not scalable, and 2 adds additional client complexity. 3 is what I'd like to see, but I don't think it can be done using a typical reliable messaging system such as AMQP. I hope I'm wrong though. Options 1) and 3) are kind of interchangeable and it depends on the circumstances which one to use. Both are possible. I'm glad to inform you that you are indeed kind of wrong. AMQP is used for this kind of things through private unnamed temporary reply queues. An example: Program A needs all diffs since time T so it creates a private reply queue the server generates a temporary name and then it just sends a message to the Query queue with the routing key osm.query.osc and the payload is just the timestamp (or any extra options one might think of) with the replyTo[3] field set to the previously created private queue. Some (might be a different one depending on the query kind, could for example decide dynamically to use an API or a XAPI reply handler depending on the request) process reads the message and begins sending the reply on the private queue which is automatically destroyed at the end. Okay, that sounds workable. Perhaps a little more complicated than a HTTP or raw socket request, but that's not necessarily bad. Something to note about the current replication mechanism is that it doesn't use any transactional capabilities other than creating files in a particular order. All replication state tracking is client side where transactions are actually occurring (eg. writing to a database, updating a planet file, etc) which keeps the server highly scalable and agnostic of client reliability. I didn't mean to say that what you're doing is wrong or bad in any way! I'm sorry if it came across as if I want to disregard everything you have done with Osmosis. The replication diffs are a huge step forward. It's all fine :-) I'm more than happy to see alternative ideas out there. And to be perfectly honest, if a better system comes out of it and Osmosis replication can be retired then I'm happy to reclaim a bit of my life back ;-) I don't know how you'd hook into the Ruby API effectively and reliably. You can't just wait for changeset closure events because changesets can remain open for large periods of time. You really want to be replicating data as soon as possible after it becomes available to API queries. This may mean receiving notification about every single entity as it is created, modified or deleted from the db, but this will result in huge
Re: [OSM-dev] TRAPI status
On Thu, Mar 11, 2010 at 10:10 PM, Lars Francke lars.fran...@gmail.comwrote: I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. I've also spent a fair bit of time thinking about this type of thing. When I first started work on the replication diffs I had in mind a server-side daemon (using Osmosis internally) that would push changes to all connected clients. It would allow a client to connect, specify which replication number it was up to, receive all updates in a single stream, then continue to receive live changes as they occurred. These are things we talked about at the recent FOSSGIS conference. I planned to do a more detailed write-up but I don't know when I'll get to that so this is not finished. We too agreed that some kind of streaming/subscription to changesets would be a good idea. Our focus though was on the german dev servers so that not everyone would need to write an .osc parser and download the changeset files etc. Cool, sounds interesting. I'll take a look if you get something written up. But I would really like to see it happen :-) I currently use AMQP (RabbitMQ) for message processing and it works very well. It is very flexible and it'd be easy to extend it with a PubSubHubBub or XMPP output. Mitja (of OpenStreetBugs) proposed just yesterday a filter that filters changes by the tags/changes involved so it would be very easy to subscribe to only the events you are interested in. This can be implemented in just a few lines of code. While we only thought about doing this on the german dev servers I've since gotten multiple requests/questions and suggestions that this should be integrated into the main OSM site. All that'd be needed would be a call in the Ruby API that sends a message (asynchronously, very fast) once a change has been made. This would make the generation of the diff files a lot easier and everything more flexible. I haven't yet asked anyone if this would be a possible. I know that this isn't the right topic (although the TRAPI could also use this system) but I wanted to take the opportunity to inform about our ideas. Thanks for the info. I'll add a few comments (because I can't help myself ;-). Most OSM systems tend to have a large number of disorganised and uncontrolled clients. Does this work well with the AMQP paradigm? In other words, does it take administrative overhead to register new subscriptions to a queue? What happens if large numbers of subscriptions are created then the clients disappear? Is AMQP targetted at a world where the clients are relatively controlled and small in number? It's important to minimise administration overhead where possible. Clients will experience outages whether that be due to a network problem, server reboot, or just not running a system 24x7. Presumably they need a way to catch up on missed events. There are a few options: 1. The server holds events for clients until they become available again, 2. The client catches up using an out of band mechanism (eg. downloads diffs directly), or 3. The client can request that the server begin sending data from a specific point. I think that only options 1 and 2 are possible using AMQP. 1 is not scalable, and 2 adds additional client complexity. 3 is what I'd like to see, but I don't think it can be done using a typical reliable messaging system such as AMQP. I hope I'm wrong though. Something to note about the current replication mechanism is that it doesn't use any transactional capabilities other than creating files in a particular order. All replication state tracking is client side where transactions are actually occurring (eg. writing to a database, updating a planet file, etc) which keeps the server highly scalable and agnostic of client reliability. I don't know how you'd hook into the Ruby API effectively and reliably. You can't just wait for changeset closure events because changesets can remain open for large periods of time. You really want to be replicating data as soon as possible after it becomes available to API queries. This may mean receiving notification about every single entity as it is created, modified or deleted from the db, but this will result in huge numbers of events which will be difficult to process in an efficient manner. The current replication solves all of this nicely by querying for data as it is committed to the database. After data is committed by the API, it becomes available to both the API and replication at the same time. The downside with the current mechanism is that it has to poll the db for changes, however the transaction check query is exceptionally fast which makes the polling overhead quite low. I also think
Re: [OSM-dev] TRAPI status
I currently use AMQP (RabbitMQ) for message processing and it works very well. It is very flexible and it'd be easy to extend it with a PubSubHubBub or XMPP output. Mitja (of OpenStreetBugs) proposed just yesterday a filter that filters changes by the tags/changes involved so it would be very easy to subscribe to only the events you are interested in. This can be implemented in just a few lines of code. While we only thought about doing this on the german dev servers I've since gotten multiple requests/questions and suggestions that this should be integrated into the main OSM site. All that'd be needed would be a call in the Ruby API that sends a message (asynchronously, very fast) once a change has been made. This would make the generation of the diff files a lot easier and everything more flexible. I haven't yet asked anyone if this would be a possible. I know that this isn't the right topic (although the TRAPI could also use this system) but I wanted to take the opportunity to inform about our ideas. Thanks for the info. I'll add a few comments (because I can't help myself ;-). That's exactly what I was looking/hoping for :) Most OSM systems tend to have a large number of disorganised and uncontrolled clients. Does this work well with the AMQP paradigm? In other words, does it take administrative overhead to register new subscriptions to a queue? What happens if large numbers of subscriptions are created then the clients disappear? Is AMQP targetted at a world where the clients are relatively controlled and small in number? It's important to minimise administration overhead where possible. I'll try to answer those in order. Subscriptions in AMQP are defined client side[1] so there is _no_ administrative overhead at all. Clients only need to know which exchange/queue they should bind to and that's just a string they need to know. Queues can be declared in different ways, one way is to declare it as autoDelete[2]: true if we are declaring an autodelete queue (server will delete it when no longer in use). Again no administrative overhead. It just disappears when no longer used so no messages are routed there. So depending on the use case there are multiple options. I have no huge real-world experience with AMQP but I believe it is targeted at both worlds: Small and controlled and large and uncontrolled. The structure they have found seems to work really well in any case. Since the installation of the RabbitMQ server I did not have to touch it again. The only thing one _could_ do is to add some kind of user management (currently everyone is allowed to do everything for example send fake messages[4]) to allow only certain users to publish to certain exchanges but that are two lines in the admin console. Clients will experience outages whether that be due to a network problem, server reboot, or just not running a system 24x7. Presumably they need a way to catch up on missed events. There are a few options: 1. The server holds events for clients until they become available again, 2. The client catches up using an out of band mechanism (eg. downloads diffs directly), or 3. The client can request that the server begin sending data from a specific point. I think that only options 1 and 2 are possible using AMQP. 1 is not scalable, and 2 adds additional client complexity. 3 is what I'd like to see, but I don't think it can be done using a typical reliable messaging system such as AMQP. I hope I'm wrong though. Options 1) and 3) are kind of interchangeable and it depends on the circumstances which one to use. Both are possible. I'm glad to inform you that you are indeed kind of wrong. AMQP is used for this kind of things through private unnamed temporary reply queues. An example: Program A needs all diffs since time T so it creates a private reply queue the server generates a temporary name and then it just sends a message to the Query queue with the routing key osm.query.osc and the payload is just the timestamp (or any extra options one might think of) with the replyTo[3] field set to the previously created private queue. Some (might be a different one depending on the query kind, could for example decide dynamically to use an API or a XAPI reply handler depending on the request) process reads the message and begins sending the reply on the private queue which is automatically destroyed at the end. Something to note about the current replication mechanism is that it doesn't use any transactional capabilities other than creating files in a particular order. All replication state tracking is client side where transactions are actually occurring (eg. writing to a database, updating a planet file, etc) which keeps the server highly scalable and agnostic of client reliability. I didn't mean to say that what you're doing is wrong or bad in any way! I'm sorry if it came across as if I want to disregard everything you have done with Osmosis. The
Re: [OSM-dev] TRAPI status
On Wed, Mar 10, 2010 at 11:21 PM, Bernhard zwischenbrugger b...@datenkueche.com wrote: Brett Henderson schrieb: On Wed, Mar 10, 2010 at 7:30 PM, Peter Körner osm-li...@mazdermind.de mailto:osm-li...@mazdermind.de wrote: I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. What about a XMPP groupchat for changesets? It's easy: login... presence to=o...@conference.osm.org/osm message to=o...@conference.osm.org type=groupchat id=58 changeset /changeset /message The OSM Server should send all changesets to this groupchat. If somebody wants the changesets he can connect to the groupchat and will get the changesets. If the messages get an incrementing serial number, it should be possible to load changesets if they are missing because of whatever. I don't know if a jabber server is able to handle the load. But facebook also uses ejabberd (free software) and it's working. If somebody wants to implement pubsub he can use the input from groupchat and load it to pubsub. That should be easy. The biggest problem I see with XMPP is keeping in sync. If a client misses one or more changesets it has to separately retrieve the missing changesets from the planet server to catch up. In the meantime it has to remain on the XMPP channel and hold the received changesets locally until it has caught up and can apply them. But don't let me dissuade you. If you think it's a good idea then you can implement something that feeds off the existing changesets. If it works well and people find it useful we can look at ways of incorporating it into the existing replication setup. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. I've also spent a fair bit of time thinking about this type of thing. When I first started work on the replication diffs I had in mind a server-side daemon (using Osmosis internally) that would push changes to all connected clients. It would allow a client to connect, specify which replication number it was up to, receive all updates in a single stream, then continue to receive live changes as they occurred. These are things we talked about at the recent FOSSGIS conference. I planned to do a more detailed write-up but I don't know when I'll get to that so this is not finished. We too agreed that some kind of streaming/subscription to changesets would be a good idea. Our focus though was on the german dev servers so that not everyone would need to write an .osc parser and download the changeset files etc. Several issues preventing this are: 1. Writing it. I don't have much time to do it therefore I went with the simplest approach that would work at the time. I'm spending some time on something like this as part of the OSMdoc rewrite. 2. Cacheability. A push approach is non-cacheable and therefore would increase the amount of bandwidth required. I suspect diff updates are small compared with planet downloads though so perhaps this is a non-issue. I'd just don't care about that. We have enough resources. So I agree that this is a non-issue. But I would really like to see it happen :-) I currently use AMQP (RabbitMQ) for message processing and it works very well. It is very flexible and it'd be easy to extend it with a PubSubHubBub or XMPP output. Mitja (of OpenStreetBugs) proposed just yesterday a filter that filters changes by the tags/changes involved so it would be very easy to subscribe to only the events you are interested in. This can be implemented in just a few lines of code. While we only thought about doing this on the german dev servers I've since gotten multiple requests/questions and suggestions that this should be integrated into the main OSM site. All that'd be needed would be a call in the Ruby API that sends a message (asynchronously, very fast) once a change has been made. This would make the generation of the diff files a lot easier and everything more flexible. I haven't yet asked anyone if this would be a possible. I know that this isn't the right topic (although the TRAPI could also use this system) but I wanted to take the opportunity to inform about our ideas. Cheers, Lars ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. I've also spent a fair bit of time thinking about this type of thing. When I first started work on the replication diffs I had in mind a server-side daemon (using Osmosis internally) that would push changes to all connected clients. It would allow a client to connect, specify which replication number it was up to, receive all updates in a single stream, then continue to receive live changes as they occurred. Unless there is some kind of filtering involved, i don't see the advantage over the current replication approach (despite of bandwidth, maybe). But if I can subscribe to all changes in a bbox or to all changes on ways with a highway-tag, then there's a real big benefit, as i don't have to deal with all changes (which is quite a huge amount of data) but only with those things I'm interested in. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Wed, Mar 10, 2010 at 7:30 PM, Peter Körner osm-li...@mazdermind.dewrote: I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. I've also spent a fair bit of time thinking about this type of thing. When I first started work on the replication diffs I had in mind a server-side daemon (using Osmosis internally) that would push changes to all connected clients. It would allow a client to connect, specify which replication number it was up to, receive all updates in a single stream, then continue to receive live changes as they occurred. Unless there is some kind of filtering involved, i don't see the advantage over the current replication approach (despite of bandwidth, maybe). The main advantage is a huge reduction in latency by elimination of most round trips. The longer term advantage this leads to is allowing the current minute replication to drop to even lower intervals (eg. 5 seconds). But I'm not sure how useful this would be in practice. But if I can subscribe to all changes in a bbox or to all changes on ways with a highway-tag, then there's a real big benefit, as i don't have to deal with all changes (which is quite a huge amount of data) but only with those things I'm interested in. I could see it being a useful tool. It would require a database of some kind to allow these types of queries to be performed. But it could be done outside the core infrastructure which reduces the number of hurdles to overcome. All it requires is somebody willing to have a crack at implementing it ;-) Cheers, Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
Brett Henderson schrieb: On Wed, Mar 10, 2010 at 7:30 PM, Peter Körner osm-li...@mazdermind.de mailto:osm-li...@mazdermind.de wrote: I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. What about a XMPP groupchat for changesets? It's easy: login... presence to=o...@conference.osm.org/osm message to=o...@conference.osm.org type=groupchat id=58 changeset /changeset /message The OSM Server should send all changesets to this groupchat. If somebody wants the changesets he can connect to the groupchat and will get the changesets. If the messages get an incrementing serial number, it should be possible to load changesets if they are missing because of whatever. I don't know if a jabber server is able to handle the load. But facebook also uses ejabberd (free software) and it's working. If somebody wants to implement pubsub he can use the input from groupchat and load it to pubsub. That should be easy. Bernhard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Fri, Mar 5, 2010 at 2:12 AM, Ian Dees ian.d...@gmail.com wrote: On Thu, Mar 4, 2010 at 9:05 AM, Andy Allan gravityst...@gmail.com wrote: On Thu, Mar 4, 2010 at 3:52 AM, Blars Blarson openstreetmap-...@scd.debian.net wrote: This is just to let you all know TRAPI development has been suspended and may never resume. That's a great shame. Would it be possible for you to commit to svn the work-in-progress, in case anyone wants to finish off your hard work at some point in the future? Also, forgive me if this shows my ignorance of TRAPI, but do you think it would be feasible to rework the import to use osmosis for the replication diff fetching? I was thinking that a custom output plugin could be created to write out the osm data in whichever format was needed for the rest of the TRAPI update process. I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. I've also spent a fair bit of time thinking about this type of thing. When I first started work on the replication diffs I had in mind a server-side daemon (using Osmosis internally) that would push changes to all connected clients. It would allow a client to connect, specify which replication number it was up to, receive all updates in a single stream, then continue to receive live changes as they occurred. Several issues preventing this are: 1. Writing it. I don't have much time to do it therefore I went with the simplest approach that would work at the time. 2. Cacheability. A push approach is non-cacheable and therefore would increase the amount of bandwidth required. I suspect diff updates are small compared with planet downloads though so perhaps this is a non-issue. 3. TCP port availability. The UCL servers only have limited ports accessible to the outside world. This would probably have to sit behind an Apache web server and therefore be HTTP based. 4. Additional administration overhead. For the last few months I have spent literally zero time looking after the replication updates. I can't remember the last time I logged into the services server. Unless somebody else is stepping in and helping out without me being aware, the current mechanism is very reliable. Occasional db or network errors occur causing the cron-based replication jobs to fail, but they recover and pick up where the left off next time things become available again. In summary, the current approach isn't perfect and could certainly be improved. But unless somebody has the time and inclination to write it themselves, go to the trouble of incorporating it into the existing infrastructure, resolve the inevitable issues that arise, and be prepared to administer it on an ongoing basis, it probably won't change any time soon. But I would really like to see it happen :-) Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Thu, Mar 4, 2010 at 3:52 AM, Blars Blarson openstreetmap-...@scd.debian.net wrote: This is just to let you all know TRAPI development has been suspended and may never resume. That's a great shame. Would it be possible for you to commit to svn the work-in-progress, in case anyone wants to finish off your hard work at some point in the future? Also, forgive me if this shows my ignorance of TRAPI, but do you think it would be feasible to rework the import to use osmosis for the replication diff fetching? I was thinking that a custom output plugin could be created to write out the osm data in whichever format was needed for the rest of the TRAPI update process. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Thu, Mar 4, 2010 at 9:05 AM, Andy Allan gravityst...@gmail.com wrote: On Thu, Mar 4, 2010 at 3:52 AM, Blars Blarson openstreetmap-...@scd.debian.net wrote: This is just to let you all know TRAPI development has been suspended and may never resume. That's a great shame. Would it be possible for you to commit to svn the work-in-progress, in case anyone wants to finish off your hard work at some point in the future? Also, forgive me if this shows my ignorance of TRAPI, but do you think it would be feasible to rework the import to use osmosis for the replication diff fetching? I was thinking that a custom output plugin could be created to write out the osm data in whichever format was needed for the rest of the TRAPI update process. I started working on a streaming XML output plugin for Osmosis. I was intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP (so that you get a 1-min delayed IM when someone changes something in your bbox). Anyway, TRAPI could use this same plugin to apply updates to their database. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
I don't know what the big fuss is about here. It is indeed sad that the development stops, but it was not the /only/ XAPI/API server out there. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Thu, Mar 4, 2010 at 10:05 AM, Andy Allan gravityst...@gmail.com wrote: On Thu, Mar 4, 2010 at 3:52 AM, Blars Blarson openstreetmap-...@scd.debian.net wrote: This is just to let you all know TRAPI development has been suspended and may never resume. That's a great shame. Would it be possible for you to commit to svn the work-in-progress, in case anyone wants to finish off your hard work at some point in the future? Also, forgive me if this shows my ignorance of TRAPI, but do you think it would be feasible to rework the import to use osmosis for the replication diff fetching? I was thinking that a custom output plugin could be created to write out the osm data in whichever format was needed for the rest of the TRAPI update process. Cheers, Andy This was already done, and both TRAPI servers that are part of the t...@h load balancer use exactly this process. This is the ugly hack that Blars refers to in his first email. I can send more info if anyone is interested in setting it up. I also plan on updating the wiki with this info as soon as I have a chance. -Jeremy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
In article c4193f8c1003040705n233ad983s51cda0e4fe474...@mail.gmail.com you write: On Thu, Mar 4, 2010 at 3:52 AM, Blars Blarson openstreetmap-...@scd.debian.net wrote: This is just to let you all know TRAPI development has been suspended and may never resume. That's a great shame. Would it be possible for you to commit to svn the work-in-progress, in case anyone wants to finish off your hard work at some point in the future? Without the documentation that hasn't yet been written, I'm not sure it would do anyone much good. Also, forgive me if this shows my ignorance of TRAPI, but do you think it would be feasible to rework the import to use osmosis for the replication diff fetching? It shouldn't be much more than 2-3 time as much work as writing the code to fetch the replication diffs properly. That would also require java and a lot of overhead I refuse to add to TRAPI. I should have been clear that my lack of time was the primary reason I stopped developing TRAPI. Coupled with my not wanting to drop everything for an immediate write of code for replication diffs, and thinking that I might be able to get them fixed so they don't double the number of files needed to be fetched, so I just stopped working on TRAPI. Due to my slow internet and systems, it would take me several weeks to do a database import. -- Blars Blarson blar...@scd.debian.net With Microsoft, failure is not an option. It is a standard feature. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
In article 20100304153805.ga12...@hydra.gt.owl.de The files you need more are the state files which are just 20 byte of status information. Plus several k-bytes and several round-trip times of overhead to fetch them. Sometime I think the people who design this stuff have never been on a slow inteernet connection. -- Blars Blarson blar...@scd.debian.net With Microsoft, failure is not an option. It is a standard feature. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
On Thu, Mar 04, 2010 at 01:52:22PM -0800, openstreetmap-...@scd.debian.net wrote: In article 20100304153805.ga12...@hydra.gt.owl.de The files you need more are the state files which are just 20 byte of status information. Plus several k-bytes and several round-trip times of overhead to fetch them. Sometime I think the people who design this stuff have never been on a slow inteernet connection. When running a TRAPI and beeing able to download a multi-gigabyte planet file + downloading a multi-kilobyte diffs per minute would suggest one has a half way speedy connection. Flo -- Florian Lohoff f...@zz.de Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] TRAPI status
I can send more info if anyone is interested in setting it up. I also plan on updating the wiki with this info as soon as I have a chance. -Jeremy I've gone ahead and updated the wiki. Let me know if anything doesn't make sense. http://wiki.openstreetmap.org/wiki/Trapi -Jeremy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] TRAPI status
As some of you know, I was working on a fairly major rewrite of TRAPI. Due to personal issues, the amount of time I spent on OSM dropped dramaticly, and almost all of this was spent on the rewrite. (I was not keeping up with the mailing lists.) This work was nearing completion when the minute diffs became unavailable. At that point, I just stopped work on TRAPI in frustration and pretty much with OSM. The replication diffs are poorly documented, require fetching twice as many files, have no equevelent of the daily and hourly sets, and the only way to figure out which one to fetch for a specific time is a binary search of fetching many files. I'm not sure a slower system with a slow internet connection, such as I'm using for TRAPI development, could ever catch up using replication diffs. I am aware of the ugly hack others have added to the old version of TRAPI to get it to use replication diffs. There is zero chance this will ever become part of my version of trapi. This is just to let you all know TRAPI development has been suspended and may never resume. -- Blars Blarson blar...@scd.debian.net With Microsoft, failure is not an option. It is a standard feature. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
