As often with such topics, the movement of the discussion is so fast and varied that it's hard to get a solid hold on any of it, but here's my .02, for what it's worth. And I apologize in advance for the length of this email, since I'm trying to respond to both the whole threat, and specific comments made.
1. Yes, there's a problem I think we're all in general agreement that the status quo won't work for a whole lot longer. I've been saying this with increasing volume since I joined the project. We're growing exponentially and I fear that the exponential wall is going to smack us in the face sooner than we'd like. Having a problem isn't a terrible thing though, so let's not let emotions run wild. 2. Identifying broad, sweeping problems without a proposed solution is silliness If we're interested in doing something other than trolling, we have to identify specific needs and problems. I think the thread is finally doing that, hence jumping in now. The main problems I see being expressed are: a) The Read API is slow b) The Write API is slow c) Machines won't grow to fit the problem d) (and I haven't seen this expressed clearly but I've only skimmed some of the emails) OSM is not geographically dispersed. I think these are all currently addressable problems 3. The future is going to be awesome, but it's not here yet. I think that in the far flung future, there will absolutely be need and utility for some kind of truly distributed mapping application that uses lots of sources. Unfortunately we have plenty of trouble running the existing OSM and trying to reconcile what we have, so at this moment, a distributed map of this type would be all noise and be of little value. The problems with this distributed approach isn't technical, it's logical, which is far harder to solve, so let's stick to the unified dataset model for the time being and the problems ahead. 4. The OSMF/ Working Groups Aren't Wizards on a Hill There seems to be a communal split in the discussion between those who think they should take up the effort and fork, and those who think the hard decisions belong to those in charge (OSMF/TWG, etc.). I reject both notions. My view on OSM forking is well known, so let's address the second point. The Technical Working Group are very smart people, but it's up to the community to come up with clever solutions to problems as well. Like other software projects, sometimes you have to take the raw material, fudge with it, and then come up with something to show. Last year at SoTM, I did a lightening talk on using MongoDB to store OSM data. I've largely been lax in working on the project, but I was addressing some of the concerns I hear being expressed in this threat. I'd love to see others working on solutions of their own, to present to the technical working group, to show them potential innovative solutions, and to be part of the way forward. 5. These are good problems to have Horizontal scaling is an interesting problem, and I think that if there were a clever coder out there who created a good solution to this that worked reliably, they could add some other features in the stack. Now let's talk about things we can do today to fix the current situation: 6. API reads can and should be made on dedicated hardware. The favored XAPI implementation is changing, and I'd bet Ian would be able (and willing, given the hardware) to add single object and history support. That's give it the ability to serve most of the read API needs. Furthermore, with a fully baked solution, it'd be easier to geographically dispurse the reads across the world when the time for that comes. I'm willing to bet that Ian's Java XAPI is faster at handling a bbox request than Rails is, and so as soon as we can move the requests to this implementation, the better, even if the hardware stayed the same. 7. Query optimisation We tend to think of our queries as being fairly straightforward, but maybe someone smart who has time could examine the logs to see if that's the case and if there's not way that a query could be optimized. I know Ian worked on query optimization in the XAPI implementation. Query optimization is difficult and relatively specialized. Nonetheless we may be able to get even more performance, and thus faster response times for the users, and lower load on the hardware. 8. Encoding, alternate client protocols, etc. We do support Gzip encoded requests and I think both Josm and Potlatch support them by default (I know Josm does for sure). Simply by supporting gzip, I've seen a 30% speedup. Any editor which doesn't use gzip encoding should be fixed to do so. We may want to look at other client-server optimisations to make in order to shorten the time that the request is delivered to the client as well. 9. Data formats We all love XML but I know for myself that XML is expensive to encode to and decode properly and even a SAX parser has significant overhead. If you think about a client needing to consume large data resources, I'm wondering if there aren't more efficient data formats we could be using, including something as simple as JSON, which might result in, again, faster updates to the browser. Longer term: 10. Geographic/Network optimised mirrors. Once an update mechanism is set in stone, the OSMF could federate Read-Only mirrors geographically. 11. I think we can do distributed writes. I think we can do distributed writes. I also know distributed writes are unpopular because of the issues they could create, but I think if we make three changes, we could do it. a) Replace serial object IDs with UUIDs Using serial interger IDs means that servers need to keep a lot of state between them. If node IDs are effectively random, this problem goes away. b) Replace the changeset metaphor with a transaction metaphor Everyone who comes to the project wants to think that a changeset is atomic. It's confusing. I think a future OSM protocol should replace this with a new metaphor which would be atomic, and wouldn't be able to span multiple requests. One request would be a single transaction, and we'd ensure that these transactions would be atomic, even if they're slow. But even this may be largely mitigated by my third suggestion. c) Provide an update mechanism for the editors While offline editing is used by folks in HOT (usually to add data!), the vast majority of editors are editing while online. So to reduce the chances of conflict, it would be best if the clients "checked in" to an area, and were then sent updates of that area. How might this work in practice? I open my editor and select an area, and then the area I'm looking at in my screen is represented by a bbox which I sent to the server in a long-poll request. If data is updated in that area, the server sends me the changes and I make a new request. By doing this, my client has a chance to catch conflicts before they're submitted to the server. If there is a conflict in my submitted chances, I'm able to address it locally. This doesn't absolutely ensure that there are no conflicts, but it could reduce the chances. Right now this might be too many requests for us to handle, but once we have more servers in place, I think this would be entirely doable, and feel natural and collaborative in the way that Google Docs or Etherpad does. I apologise for a long mail, and I may be just adding to the trolls, but I think that with modest changes to our systems, we can take ourselves pretty far, and with some moderate changes, be in a very good position to tackle these scaling issues. - Serge _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
