Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Stefan de Konink wrote: Yup; I know. But when Gert told me it costed 2x 150 euro + a place to sleep. I was like... come on I can fly to Russia for that money. So we have now 6 hours of other development (paid) time left :) This might come as a shock, but sometimes 300€ plus 30€ for a hotel is worth meeting people who code physically and in real life. It helps reduce misunderstandings and mail flamewars, and increases empathy, trust and helps coordinating things. :-) Especially if you have a solution that is superior to whatever there is right now. It's not going to install itself on the OSM servers, even if the code is publicaly available. spaetz ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Sun, Nov 09, 2008 at 04:53:46PM +, Shaun McDonald wrote: Actually you will find that having a limit of 2000 nodes in a way will make osm more scaleable. It will make it easier to use osm data on mobile devices, and on devices with low memory. As it is if you try and edit a large way in one of the current editors, you will find it very slow. That’s why the ability to deal with partial ways was suggested. Simon -- A complex system that works is invariably found to have evolved from a simple system that works.—John Gall signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Sun, Nov 09, 2008 at 10:30:25AM +, Andy Allan wrote: On Sun, Nov 9, 2008 at 3:48 AM, Simon Ward [EMAIL PROTECTED] wrote: Stop making it more complex by introducing arbitrary limits because of technical issues, especially ones that go away when you move to better hardware or software. Start making things scalable. Your attitude stinks. Thanks, the eau de toilette is working, excellent! I'm sat here for the second day of my weekend, along with another 8 people (which makes up a large chunk of our development community) working on our free time on the code and running the servers. You realise what that means, right? Actually doing some work, not just back seat driving? Right, and I’m here thinking how absurd the development is going. I commented on it because I thought it was absurd. I’ve been looking to get into OSM development more, sorry I’m not there yet, but *your* attitude just puts me right off. I absolutely love the work you (and Dave, and others?) have done on the cycle map, but if you can’t accept criticism graciously I can’t help but think you value the project less than your self esteem. Keep your thoughts to yourself in future, unless you have something more constructive to say, or preferably code to contribute. Not everyone’s as hardcore a developer as you. Simon -- A complex system that works is invariably found to have evolved from a simple system that works.—John Gall signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Sun, Nov 09, 2008 at 01:57:51PM +0100, Stefan de Konink wrote: Simon: I hope you were not turned of to help around with OSM. Thanks. It won’t be that easy to dissuade me from helping with OSM development, fortunately, but I really need to get a feel for the code first, and have been on the lookout for something to get my teeth into. I’m not really a Ruby person though, so unlikely to be the OSM server stuff any time soon. I’m more reading up on SVG + XSLT + Javascript to work with maps because those are things I work with at work. Simon -- A complex system that works is invariably found to have evolved from a simple system that works.—John Gall signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Simon Ward wrote: I think I agree with this. Contributors or users of the data shouldn’t have to care at all about how big their ways (or other objects) can be. One of the main aims for OSM is that it is easy to contribute data, right? Stop making it more complex by introducing arbitrary limits because of technical issues, especially ones that go away when you move to better hardware or software. Start making things scalable. This probably means having partial ways as suggested. Unfortunately our number of developers has never scaled particularly well. So if you want it done, you need to do it, potentially including the work on the principal clients (I guess JOSM, Potlatch, Merkaartor, Osmosis?). Otherwise pragmatism rules and we will continue to balance the ideal solution against the achievable one given the priorities of our limited number of developers. In other words, the imperative is not the correct mood to use. ;) cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Hi, Stefan de Konink wrote: Your attitude (and body) stinks more. Someone as articulate as that should immediately made honorary chairman of the OSM foundation! Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Sun, Nov 9, 2008 at 12:57 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Andy Allan wrote: Your attitude stinks. Your attitude (and body) stinks more. [...] Now go on stinking, because I highly doubt that you took the time to take a shower. Ad hominem attacks are pointless. I'm not going to respond to any of the rest of your post. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Andy Allan wrote: On Sun, Nov 9, 2008 at 12:57 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Andy Allan wrote: Your attitude stinks. Your attitude (and body) stinks more. [...] Now go on stinking, because I highly doubt that you took the time to take a shower. Ad hominem attacks are pointless. I'm not going to respond to any of the rest of your post. I rest my case :) You just prove that your flaming to Simon would never result in a discussion about the subject. Likewise this flaming is not going to result in a discussion about the subject. Thanks for that. Wat gij niet wilt dat u geschiedt, doe dat dan ook een ander niet.[1] Stefan [1] http://en.wikipedia.org/wiki/Ethic_of_reciprocity ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Andy Allan wrote: Your attitude stinks. Your attitude (and body) stinks more. I proposed the original, he was commenting on! I sat all night programming a binary encoding for OSM! Go look on the same 'weekend-wiki page' for my name, hence I didn't want to spend 2x 150 euro to sit somewhere else programming. So if you want to have someone to say how GOOD you are. YOU ARE SO GOOD TO PROVIDE US SOME RUBY IMPLEMENTATION OF 0.6! Now go on stinking, because I highly doubt that you took the time to take a shower. My god, this is so pathetic, flaming on people that actually care about arguments given, being trolled over some code monkey that is able to type Ruby. As you might not noticed, if this was a programmer your arguments were *kind of* counter productive to get him coding. Simon: I hope you were not turned of to help around with OSM. Making things scalable is my first priority. Hence my C implementation of what they are trying to build. I'll not start with my 0.6 until the specs are finished and someone is actually using it. But don't think you have to wait very long for an concurrent work that has limits. I don't even imply bbox limits on my code. Network traffic has to be reduced, and this is currently what I am making. Yours Sincerely, Stefan de Konink ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On 9 Nov 2008, at 03:48, Simon Ward wrote: On Fri, Oct 31, 2008 at 12:02:22PM +0100, Stefan de Konink wrote: You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. I think I agree with this. Contributors or users of the data shouldn’t have to care at all about how big their ways (or other objects) can be. One of the main aims for OSM is that it is easy to contribute data, right? Stop making it more complex by introducing arbitrary limits because of technical issues, especially ones that go away when you move to better hardware or software. Start making things scalable. This probably means having partial ways as suggested. Actually you will find that having a limit of 2000 nodes in a way will make osm more scaleable. It will make it easier to use osm data on mobile devices, and on devices with low memory. As it is if you try and edit a large way in one of the current editors, you will find it very slow. Shaun smime.p7s Description: S/MIME cryptographic signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Stefan de Konink wrote: Go look on the same 'weekend-wiki page' for my name, hence I didn't want to spend 2x 150 euro to sit somewhere else programming. I looked. I saw this: Please note that for a limited number of people (say 4-5) a contribution in the travel costs may be obtained via OSM-NL -- GertGremmen 11:10, 5 November 2008 (UTC) cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Richard Fairhurst wrote: Stefan de Konink wrote: Go look on the same 'weekend-wiki page' for my name, hence I didn't want to spend 2x 150 euro to sit somewhere else programming. I looked. I saw this: Please note that for a limited number of people (say 4-5) a contribution in the travel costs may be obtained via OSM-NL -- GertGremmen 11:10, 5 November 2008 (UTC) Yup; I know. But when Gert told me it costed 2x 150 euro + a place to sleep. I was like... come on I can fly to Russia for that money. So we have now 6 hours of other development (paid) time left :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Fri, Oct 31, 2008 at 12:02:22PM +0100, Stefan de Konink wrote: You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. I think I agree with this. Contributors or users of the data shouldn’t have to care at all about how big their ways (or other objects) can be. One of the main aims for OSM is that it is easy to contribute data, right? Stop making it more complex by introducing arbitrary limits because of technical issues, especially ones that go away when you move to better hardware or software. Start making things scalable. This probably means having partial ways as suggested. Simon -- A complex system that works is invariably found to have evolved from a simple system that works.—John Gall signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Frederik Ramm wrote: This definitely has to stop. We need to (a) find all ways with more than a few thousand nodes and break them down, and (b) educate users that they shouldn't do such evil things. Imagine the poor sod who opens a little rectangle in JOSM just to find he has to wait for ages to download 40k nodes! This slows down so many things. I've just had the [EMAIL PROTECTED] client crash on me because of a way of some 4700 nodes (which wasn't even in the tile itself). Memory consumption went up to 1GB for inkscape and then crash. So I split the way up in maximum 1000 nodes and memoryuse was no more than 300MB. Seems to be a quite linear relation between number of nodes and Inkscape memory usage. I'm all in favour of trying to dissuade people from creating such long ways. Is it also an option to ask the editor developers (Potlatch, JOSM, Merkaartor) to invoke a limit? BTW: this was a border: Northwest Arctic Borough from USGS data, so most likely an automated import and would not be fixed by my suggestion above. Maarten ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Sun, 2 Nov 2008, Maarten Deen wrote: I've just had the [EMAIL PROTECTED] client crash on me because of a way of some 4700 nodes (which wasn't even in the tile itself). Memory consumption went up to 1GB for inkscape and then crash. So I split the way up in maximum 1000 nodes and memoryuse was no more than 300MB. Seems to be a quite linear relation between number of nodes and Inkscape memory usage. I guess you don't want to make a relation between your XSLT processor and memory usage? That Inkscape is not optimal for big documents was clear all along. I still think it would be interesting if you could try the same (think still not using SVG) in Xara. Next to this; you could also solve the problem by not feeding Inkscape such big documents, instead try to feed inscape dynamically smaller chunks. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On 31 Oct 2008, at 07:36, Frederik Ramm wrote: Stefan, Having a binary XML format is another approach. And would again significantly reduce the communication delay between client and api. All your comments seem to assume that we have an arbitrary amount of time an manpower to change the API plus all clients. This is a misconception, and your comments are not helpful without a workable transition strategy. +1 Bye Frederik -- Frederik Ramm ## eMail [EMAIL PROTECTED] ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev Best Steve ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On 31 Oct 2008, at 04:07, Dave Stubbs wrote: On Fri, Oct 31, 2008 at 10:44 AM, Frederik Ramm [EMAIL PROTECTED] wrote: Hi, Florian Lohoff wrote: Its 2^15 because it signed - and yes - somebody managed to get abovE: This definitely has to stop. We need to (a) find all ways with more than a few thousand nodes and break them down, and (b) educate users that they shouldn't do such evil things. Imagine the poor sod who opens a little rectangle in JOSM just to find he has to wait for ages to download 40k nodes! This slows down so many things. (And what's more, once someone creates a way with 50.001 nodes, no bounding box containing even one node of that way will be downloadable through the API.) I know that shortcomings in the renderers still make it attractive for mappers to create giant polygons but we cannot allow this to get out of hand. I'm all for an API hard limit of 1000 nodes in a way. That wouldn't impact any normal stuff, but would put a stop to megaways pretty quickly without the need for a re-education programme :-) I agree, and I think it's awesome we have this problem - better having people stretching the API than not using it. Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev Best Steve ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Fri, 31 Oct 2008, Frederik Ramm wrote: Any comments? You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On 31 Oct 2008, at 10:44, Frederik Ramm wrote: Hi, Florian Lohoff wrote: Its 2^15 because it signed - and yes - somebody managed to get abovE: This definitely has to stop. We need to (a) find all ways with more than a few thousand nodes and break them down, and (b) educate users that they shouldn't do such evil things. Imagine the poor sod who opens a little rectangle in JOSM just to find he has to wait for ages to download 40k nodes! This slows down so many things. (And what's more, once someone creates a way with 50.001 nodes, no bounding box containing even one node of that way will be downloadable through the API.) I know that shortcomings in the renderers still make it attractive for mappers to create giant polygons but we cannot allow this to get out of hand. Any comments? JonB perhaps ;-)? Should we add something to API 0.6 to block ways that have huge numbers of nodes. Looking at the map call, it would actually be downloadable, as long as the bounding box doesn't contain more than 50,000 nodes. This is because the 50,000 node check is done after getting the list of nodes in that bbox. Then the 50,000 node check happens. Then additional nodes outside the bbox for the ways that have a node in the bbox are added to the collection. Thus the api can return more than 50,000 nodes, just the nodes above 50,000 will be for ways outside the current bbox. Shaun smime.p7s Description: S/MIME cryptographic signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Fri, Oct 31, 2008 at 10:44 AM, Frederik Ramm [EMAIL PROTECTED] wrote: Hi, Florian Lohoff wrote: Its 2^15 because it signed - and yes - somebody managed to get abovE: This definitely has to stop. We need to (a) find all ways with more than a few thousand nodes and break them down, and (b) educate users that they shouldn't do such evil things. Imagine the poor sod who opens a little rectangle in JOSM just to find he has to wait for ages to download 40k nodes! This slows down so many things. (And what's more, once someone creates a way with 50.001 nodes, no bounding box containing even one node of that way will be downloadable through the API.) I know that shortcomings in the renderers still make it attractive for mappers to create giant polygons but we cannot allow this to get out of hand. I'm all for an API hard limit of 1000 nodes in a way. That wouldn't impact any normal stuff, but would put a stop to megaways pretty quickly without the need for a re-education programme :-) Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Dave Stubbs wrote: I'm all for an API hard limit of 1000 nodes in a way. That wouldn't impact any normal stuff, but would put a stop to megaways pretty quickly without the need for a re-education programme :-) +1. I've been thinking of disabling access to ways of 1000 nodes in Potlatch anyway (see http://trac.openstreetmap.org/ticket/1274), it'd be great if people couldn't create them in the first place. Recently some poor chap spent ages tracing out a lake, a well-meaning but misguided mapper came along and combined all his ways into one big one (17,000 nodes - ffs), and one bunch of timeouts later, the way ended up utterly b0rked: http://www.openstreetmap.org/user/katpatuka/diary/3589#comments cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Fri, Oct 31, 2008 at 11:02 AM, Stefan de Konink [EMAIL PROTECTED] wrote: On Fri, 31 Oct 2008, Frederik Ramm wrote: Any comments? You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. With 40k nodes in a way it takes the API about 30 seconds just to generate and serve the XML for the way, /without/ the nodes. The .osm file with no node data goes upto about 100kb. (With node data that's 5-6MB) To make a modification to the way, ie: add a tag, you have to reupload that 100kb of XML, and sit back, probably make a cup of coffee to pass the time. I don't think having a single object of this size ends up helping anybody. With relations we can add indirection to handle extremely large objects, but with small API units. It just makes everything more flexible than dealing with megalithic monoliths. Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Dave Stubbs wrote: On Fri, Oct 31, 2008 at 11:02 AM, Stefan de Konink [EMAIL PROTECTED] wrote: On Fri, 31 Oct 2008, Frederik Ramm wrote: Any comments? You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. With 40k nodes in a way it takes the API about 30 seconds just to generate and serve the XML for the way, /without/ the nodes. The .osm file with no node data goes upto about 100kb. (With node data that's 5-6MB) To make a modification to the way, ie: add a tag, you have to reupload that 100kb of XML, and sit back, probably make a cup of coffee to pass the time. Fix the API. (and the implementation if it takes you that long ;) I don't think having a single object of this size ends up helping anybody. With relations we can add indirection to handle extremely large objects, but with small API units. It just makes everything more flexible than dealing with megalithic monoliths. Imagine a way as a relation. Where every difference in way is represented as a subtree under it. In that case common tags are equal and managed at top level. In case of a partial checkout you would only fetch the subtrees (relations) that are in your bbox and their parent. Because you now know it is a partial checkout, and you have a partial subtree, based on the changes you make (the way is a connected line) every change will just be an insertion, translation or deletion of this ordered segments, hence the only thing you download. The only thing you block is the change of nodes at cut points. In this case I address all your and my problems; - common tags are not duplicated - partial ways are smaller - relations become in essence ordered linestrings, with the possibility to add tags Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Frederik Ramm wrote: (And what's more, once someone creates a way with 50.001 nodes, no bounding box containing even one node of that way will be downloadable through the API.) I think it will be actually - the 5 node limit only applies to the initial selection of nodes I think and doesn't include any dragged in from outside the bounding box by ways which cross it. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On 31 Oct 2008, at 11:51, Stefan de Konink wrote: Dave Stubbs wrote: On Fri, Oct 31, 2008 at 11:02 AM, Stefan de Konink [EMAIL PROTECTED] wrote: On Fri, 31 Oct 2008, Frederik Ramm wrote: Any comments? You are now basically working around the actual problem. Allowing partial ways in the editors for the current bbox. I think hacking and breaking ways is bad, duplicate information, missing tags upon edit etc. I think storing ways with their tags per 'new segment' is bad too; hence the reason I proposed to use an ordered relation to represent ways. With 40k nodes in a way it takes the API about 30 seconds just to generate and serve the XML for the way, /without/ the nodes. The .osm file with no node data goes upto about 100kb. (With node data that's 5-6MB) To make a modification to the way, ie: add a tag, you have to reupload that 100kb of XML, and sit back, probably make a cup of coffee to pass the time. Fix the API. (and the implementation if it takes you that long ;) Actually this isn't just an implementation problem, it is also a social limit. If 12 people were to request large highly detailed areas (and they were on a slow network connection), then you suddenly have everyone else locked out from doing bulkapi requests, until they have cleared. There are other better ways to get large amounts of data, such as mirror API servers or the planet file. With the exposure of the node/ way/relation version numbers in API 0.6 this will make it much easier to deal with mirrored api servers. Shaun smime.p7s Description: S/MIME cryptographic signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Stefan de Konink wrote: In this case I address all your and my problems; - common tags are not duplicated - partial ways are smaller - relations become in essence ordered linestrings, with the possibility to add tags But unfortunately not problem 4 - someone to actually implement it Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Stefan de Konink wrote: I have already created a server that stores in nodes/relations :) Yeah, I know you have, but you might have noticed we don't actually have editor authors coming out of our ears. Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Richard Fairhurst wrote: Stefan de Konink wrote: I have already created a server that stores in nodes/relations :) Yeah, I know you have, but you might have noticed we don't actually have editor authors coming out of our ears. Thats why legacy support is important :) But again, if you do want to improve it, let us all meet with Bart and his crew (or the opposite way around ;) and Frederik to get this wagon rolling. I think we could create something that is significantly better for the editors. Ian Dees wrote: The majority of the delay between client and API isn't in downloading the XML file, it's on the server side querying the data from the database. This same querying will need to happen no matter what kind of format is used. Hence see: change implementation ;) But from my practice where I allow xapi request over The Netherlands XML is also a problem. Files of 15MB are not fun to download. And worse: are definitely not fun to parse without something that actually has parsing performance. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Andy Allan wrote: In case of a partial checkout you would only fetch the subtrees (relations) that are in your bbox and their parent. The one subtree has 17M nodes, and I get the parent relation with the coastline tags too. No no! bbox the subtree, so you will not get 17M nodes, but only the nodes in your bbox. The subtree would be only used for tagging purposes, and maintain parent order. This would even allow non contiguous subtrees, having, for example ,a specific tag over a certain amount of nodes. No, hang on, that would be daft. I'd be better off making subtrees limited to about 1000 nodes or thereabouts, to improve the efficiency of partial checkouts. So then I would have sensibly sized subtrees (lets call them 'ways') and the master super-tree (lets call it a relation). You miss the point of rewriting ways into relations. It will reduce the code complexity and database complexity of all operations significantly. The amount of tables is reduced, that means the amount of joins are reduced. And the indices required for database searches are now more efficient. Queries therefore get shorter, no need for a specific way renderer. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Stefan, Having a binary XML format is another approach. And would again significantly reduce the communication delay between client and api. All your comments seem to assume that we have an arbitrary amount of time an manpower to change the API plus all clients. This is a misconception, and your comments are not helpful without a workable transition strategy. Bye Frederik -- Frederik Ramm ## eMail [EMAIL PROTECTED] ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Hi, Dave Stubbs wrote: I'm all for an API hard limit of 1000 nodes in a way. That wouldn't impact any normal stuff, but would put a stop to megaways pretty quickly without the need for a re-education programme :-) Fine with me. I think this can (and should) be worked on at different levels. 1. Fix all ways with more than a few thousand nodes; repeat checks regularly. Timeframe: can be done tonight. 2. Start using an area relation (or allow multiple interconnected outer parts in the existing multipolygon relation) to be able to model large areas (where the circumference has more nodes than the node limit and thus must be split). Requires a (relatively minor, I guess) modification to osm2pgsql/osmarender, and to editors which are area-aware (if unmodified, editors will display these area outline segments as lines which doesn't hurt). Timeframe: a few days. 3. Build a node limit into the API. Timeframe: could do quickly if required but could also do it for 0.6. 4. Redesign the whole data model to allow partial checkouts of ways and generally switch over to Stefan de Konik's software which is better in any respect anyway (although I haven't yet understood the big difference between checking out part of a way and checking out a whole way that is part of a grouping relation). Timeframe: Maybe sometime. Bye Frederik -- Frederik Ramm ## eMail [EMAIL PROTECTED] ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
On Fri, Oct 31, 2008 at 3:19 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: would you represent a Y-shaped road as three relations, then? one for the left fork, one for the right stem as a way-relation, then another to group them together? Yes, but with role='independent'. Otherwise the idx of the subtrees would create a pyramid shape. i don't understand what you mean by a pyramid shape. the relations would be returned in the same order, regardless of the role tag, right? and you're not merging subtrees...? when you say no need for a specific way renderer - there is still a need, as (in your example) you wouldn't want to render a bunch of camping nodes as a linear feature. you'll also want to render buildings filled. in other words: there isn't a single, specific way renderer anyway because tag inspection is required. ... snip... In the first version of the RelationOSM thing I created the same thing in the relationship code, when the api 0.5 was used. If the tag way=true was present, it would be rendered as waynds//way. exactly; you do tag inspection. doesn't that require an extra join to distinguish ways from relations of nodes? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ways with 40k nodes, was: osmosis pgsql schema
Matt Amos wrote: On Fri, Oct 31, 2008 at 3:19 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: would you represent a Y-shaped road as three relations, then? one for the left fork, one for the right stem as a way-relation, then another to group them together? Yes, but with role='independent'. Otherwise the idx of the subtrees would create a pyramid shape. i don't understand what you mean by a pyramid shape. the relations would be returned in the same order, regardless of the role tag, right? and you're not merging subtrees...? My proposal would be merging subtrees, so they can store an highway with 80km/h 100km/h zones in one relation as part of two subtrees (as optimalisation), or multiple subtrees for each zone found. when you say no need for a specific way renderer - there is still a need, as (in your example) you wouldn't want to render a bunch of camping nodes as a linear feature. you'll also want to render buildings filled. in other words: there isn't a single, specific way renderer anyway because tag inspection is required. ... snip... In the first version of the RelationOSM thing I created the same thing in the relationship code, when the api 0.5 was used. If the tag way=true was present, it would be rendered as waynds//way. exactly; you do tag inspection. doesn't that require an extra join to distinguish ways from relations of nodes? That join is always done in order to produce tags related to the node/relation. But I see your point. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
