Re: [OSM-dev] ROMA servers down - osmosis large way problem
Am Sonntag, den 28.12.2008, 18:00 -0500 schrieb Jeremy Adams: I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. An ALTER TABLE on MySQL always copies the whole table. Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Joachim Zobel wrote: Am Sonntag, den 28.12.2008, 18:00 -0500 schrieb Jeremy Adams: I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. An ALTER TABLE on MySQL always copies the whole table. Indeed, just one of the many reasons why MySQL sucks giant boulders through exceedingly small straws. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Tom Hughes wrote: Indeed, just one of the many reasons why MySQL sucks giant boulders through exceedingly small straws. The only thing worse than doing an ALTER TABLE on a large table is doing a Rails migration on it ;-) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
D Tucny wrote: That would be quite nasty, seeing node 8, but not having a line connecting it anywhere? For rendering, that would be pretty useless, for editing it would be a pain in the bum, especially if there was a node 9 somewhere else outside the bbox... Notice variation. It is an editor thing, and can made useful. To get the useful nodes 7 and 9, you'd either have to get the whole way as a seperate request and then get the nodes, as seperate requests or make guesses as to which direction and how far away those nodes are and do bbox requests... You'd need at least a one node buffer around 'in box' nodes to be of use for anything... i.e. if only node 10 is within the bbox, you'd need to return 9 and 11 too, at least, even if they are outside the bbox... in your example, that would return nodes 3, 6 and 7 too... You seem to miss the point that to see the rest of the way: 1) a request can be made with a new bbox like now is common practice if you want to edit a larger area 2) CLick on the way and do a full I don't see your buffer point. I would like to return the nodes that are exactly within the bbox. Like what is expected. To me, this feels like a significant overcomplication with no significant benefits... Far easier would be to just limit the size of ways... Hmm, I'm sure that's been mentioned already... You want to limit the ways because then you would request only 2k of nodes cluelessly instead of what you actually see? That is an over simplification. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
D Tucny wrote: I'm not seeing it... Let me first enlight you with the non-variated idea. You seem to miss the point that to see the rest of the way: 1) a request can be made with a new bbox like now is common practice if you want to edit a larger area But how do you know where the rest of the way is? Ascii art: Your viewport/bbox; --- | | | | | | | | | | --- Nodes in the way; --- | | | | |.. | | . . | | . | --- Explicit order of them; --- | | |26 | |. 3. | | . . 4 | | 1 . | --- How the editor draws it: --- | | | | |., o | | o,' ',. | | ',o | --- *see below for revisted thing* User thinks; mmm... that way is outside my bbox, lets request the area next to it. Now that area will also return a partial way; --- |---|-- | | | | |. | . | | | . . | | | | | | | ---|--- | | | -- The editor merges the result; --- | |-- | | |... . | | . . | | . . | ---| | |. | -- And now it can connect; 1, 2, 3, 4, 5, 6, 7, 8, 9 ---, | +-- | | | ,.,.-.--. | | .' '.,. , | | '. . | ---+, , | |.,| -- 2) CLick on the way and do a full So replacing one request with multiple requests... erm... Assuming you could actually find the way... If a part of your way is visible you should be able to request it by the usual 'rest' command. /ways/[id] or alternatively multible bboxes. I don't see your buffer point. I would like to return the nodes that are exactly within the bbox. Like what is expected. snip I do see some of your points. Those 'nodes' should be not marked as a node 'symbol' or 'colour' but as an endpoint. So there is explicit knowledge that this node when clicked on is viewed as a way. Otherwise it is confusing if you see all sort of artifacts ath the sides of the bbox (then it would be useful to have always at least two nodes on screen. I can see that your suggestion for modifying ways could be potentially useful to reflect changes to ways without having to retransmit the entire way, but, I don't like idea of by default not getting all the relevant information for an area... What is exactly transfered is a thing that should be tweaked indeed, even for rendering you want something that is going outside of the bbox. I guess you would like to make it: --- | | | | | ,.,o|o | o' '.,. | | '. | ,--| o And indeed this seems a much better approach. Still preferable in my eyes to be forced to be without a clue, not being able to see what you actually see due to having relevant data effectively hidden... I hope I can convince you with my ascii art ;) Either way, a maximum of 2000 nodes in a way would work in both scenarios... So no arguments from me... True but as Martijn already pointed out, there is just still stuff that should be 'one', coastallines. And if I may add, I would really like to apply algebra for geo stuff to that kind of things. That will not work if OSM limits 2k. And OSM-companies suddenly start to offer shapefiles where it actually is possible to do that kind of lookups with merged coastallines, that is a bit a turn off to me... I expect the 2000 node limit will be implemented prior to your solution being ready as the 0.6 API is already pretty much there... So, that's that... I will try to keep an open mind though, once your solution is ready, I'll evaluate it on it's merits too, but, I suspect you'll also have to make friends with some editor devs or take up editor modifying yourself to attempt to avoid noticably reducing the current usability too much... I have friends in many places ;) Thanks for your comments, they really changed my view on the matter :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
2008/12/31 Stefan de Konink ste...@konink.de D Tucny wrote: I'm not seeing it... Let me first enlight you with the non-variated idea. You seem to miss the point that to see the rest of the way: 1) a request can be made with a new bbox like now is common practice if you want to edit a larger area But how do you know where the rest of the way is? Ascii art: Your viewport/bbox; --- | | | | | | | | | | --- Nodes in the way; --- | | | | |.. | | . . | | . | --- Explicit order of them; --- | | |26 | |. 3. | | . . 4 | | 1 . | --- How the editor draws it: --- | | | | |., o | | o,' ',. | | ',o | --- *see below for revisted thing* User thinks; mmm... that way is outside my bbox, lets request the area next to it. Now that area will also return a partial way; --- |---|-- | | | | |. | . | | | . . | | | | | | | ---|--- | | | -- The editor merges the result; --- | |-- | | |... . | | . . | | . . | ---| | |. | -- And now it can connect; 1, 2, 3, 4, 5, 6, 7, 8, 9 ---, | +-- | | | ,.,.-.--. | | .' '.,. , | | '. . | ---+, , | |.,| -- Attempting to use ASCII art to illustrate my example... OK, as an example... I get a bbox that covers a 500m x 500m area... 10 ways run through that area, including 2 I'm actually interested in... right now, as long as those 10 ways have nodes within the bbox I get them all and I can see how they interact and where they go after leaving my bbox giving a useful hint as to where to expand my bbox should I want to edit a way that leaves it... With your suggestion I could see that I could have say 1 line, between 2 nodes and and 5 additional standalone nodes, the only thing I know is that the two nodes joined are part of one way and the other 5 nodes are either not part of that way, or, if they are, that there are other nodes, somewhere outside of the bbox that connects them... if I select the one visible way, as none of the nodes are highlighted, I'd know that they were not part of that way, so, they must be either standalone nodes that are not part of a way, which, with their lack of tags would suggest that they were bad edits, or, they are indeed part of a way, but for which there are no other nodes to be connected to... if we assume that they were further highlighted in your suggestion to show that they were special way nodes as opposed to non way nodes, I'd know they were part of a way, but, I don't know which way or where infact additional nodes are... I can blindly extend the bbox and cross my fingers hoping I'll get more nodes to be able to visualise part of the way, or, I guess, the editor can be made to give me a list of ways that are missing nodes so that I can download them in full manually... bbox: --- | | | | | | | | | | --- Returned data: --- | . | | .| | . . | | .| | . . | --- Order of nodes (multiple ways): --- | 8 | | 4| | 3 4 | | 4| | 2 2 | --- How the editor draws them... --- | o | | o| | oo | | o| | o o | --- How I'd expect the editor to draw them, even with partial ways... o \ \ o o \| | -|\-|-- | | |. | o--|-+--+\---.|---o o--+-+.--\--+o o--|-+--+.\---|---o | . . \ | -|---|- \ | |o | o o Expanding bboxes gets some of the nodes, but, would need to be repeated in each direction to get all of them, with no guarantees if the bbox was not expanded far enough... 2) CLick on the way and do a full So replacing one request with multiple requests... erm... Assuming you could actually find the way... If a part of your way is visible you should be able to request it by the usual 'rest' command. /ways/[id] or alternatively multible bboxes.
Re: [OSM-dev] ROMA servers down - osmosis large way problem
D Tucny wrote: I guess you would like to make it: --- | | | | | ,.,o|o | o' '.,. | | '. | ,--| o And indeed this seems a much better approach. This is what I was trying to illustrate without ascii art :) So would this be something useful/workable/better to you as editor? Either way, a maximum of 2000 nodes in a way would work in both scenarios... So no arguments from me... True but as Martijn already pointed out, there is just still stuff that should be 'one', coastallines. And if I may add, I would really like to apply algebra for geo stuff to that kind of things. That will not work if OSM limits 2k. And OSM-companies suddenly start to offer shapefiles where it actually is possible to do that kind of lookups with merged coastallines, that is a bit a turn off to me... I'm sure it wouldn't be too complex with a few recursive lookups to get an entire coastline, where you could, if you so wished merge them together and perform any, not usual for typical use, function on them, including applying algebra... I'm sure with your own copy of the planet, especially if loaded into a database it would quick and simple to achieve this too... admittedly, not as quick as if the database natively held the entire coastline as a single way, but, trivial none the less... An example of such cleverness would be the coastline checker stuff done by Martijn... The point is that 'geo databases' operate on cyclic polylines as an area. Splitting them up requires another virtual and join. Doable but I want decent arguments why to split it at data level (2k limit) opposite to user level (partial ways). (And joins are exactly expensive at OSM sizes...) So I'll try to fix a server implemementation :) And ask for some rendering help. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Happy New Year ;) Best Regards Roberto Navoni D Tucny wrote: I'm not seeing it... Let me first enlight you with the non-variated idea. You seem to miss the point that to see the rest of the way: 1) a request can be made with a new bbox like now is common practice if you want to edit a larger area But how do you know where the rest of the way is? Ascii art: Your viewport/bbox; --- | | | | | | | | | | --- Nodes in the way; --- | | | | |.. | | . . | | . | --- Explicit order of them; --- | | |26 | |. 3. | | . . 4 | | 1 . | --- How the editor draws it: --- | | | | |., o | | o,' ',. | | ',o | --- *see below for revisted thing* User thinks; mmm... that way is outside my bbox, lets request the area next to it. Now that area will also return a partial way; --- |---|-- | | | | |. | . | | | . . | | | | | | | ---|--- | | | -- The editor merges the result; --- | |-- | | |... . | | . . | | . . | ---| | |. | -- And now it can connect; 1, 2, 3, 4, 5, 6, 7, 8, 9 ---, | +-- | | | ,.,.-.--. | | .' '.,. , | | '. . | ---+, , | |.,| -- 2) CLick on the way and do a full So replacing one request with multiple requests... erm... Assuming you could actually find the way... If a part of your way is visible you should be able to request it by the usual 'rest' command. /ways/[id] or alternatively multible bboxes. I don't see your buffer point. I would like to return the nodes that are exactly within the bbox. Like what is expected. snip I do see some of your points. Those 'nodes' should be not marked as a node 'symbol' or 'colour' but as an endpoint. So there is explicit knowledge that this node when clicked on is viewed as a way. Otherwise it is confusing if you see all sort of artifacts ath the sides of the bbox (then it would be useful to have always at least two nodes on screen. I can see that your suggestion for modifying ways could be potentially useful to reflect changes to ways without having to retransmit the entire way, but, I don't like idea of by default not getting all the relevant information for an area... What is exactly transfered is a thing that should be tweaked indeed, even for rendering you want something that is going outside of the bbox. I guess you would like to make it: --- | | | | | ,.,o|o | o' '.,. | | '. | ,--| o And indeed this seems a much better approach. Still preferable in my eyes to be forced to be without a clue, not being able to see what you actually see due to having relevant data effectively hidden... I hope I can convince you with my ascii art ;) Either way, a maximum of 2000 nodes in a way would work in both scenarios... So no arguments from me... True but as Martijn already pointed out, there is just still stuff that should be 'one', coastallines. And if I may add, I would really like to apply algebra for geo stuff to that kind of things. That will not work if OSM limits 2k. And OSM-companies suddenly start to offer shapefiles where it actually is possible to do that kind of lookups with merged coastallines, that is a bit a turn off to me... I expect the 2000 node limit will be implemented prior to your solution being ready as the 0.6 API is already pretty much there... So, that's that... I will try to keep an open mind though, once your solution is ready, I'll evaluate it on it's merits too, but, I suspect you'll also have to make friends with some editor devs or take up editor modifying yourself to attempt to avoid noticably reducing the current usability too much... I have friends in many places ;) Thanks for your comments, they really changed my view on the matter :) Stefan ___ dev mailing list dev@openstreetmap.org
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Matt Amos wrote: On Mon, Dec 29, 2008 at 10:26 PM, Stefan de Konink ste...@konink.de wrote: Some people wanted to limit ways/relations to 2k of nodes. If you do allow relations to have all these extra subrelations then what are you going to solve in the end? This is enforcing a clueless limit and not solving the fundamental problem why this limit is justified; and no that has nothing to do with data storage. practicality. since we all agree that no-one wants to download a 100k+ node way, there are two immediately obvious solutions: change the API to return partial ways or disallow ways longer than some arbitrary limit. the former requires many changes on the server and client, the latter requires many fewer. The latter still requires the same client/server modifications + modifications in all current node relations. Hence it will cost more time. As I mentioned earlier, there is no need for even the concept 'way'; since you can store a relation with tag highway=whatever. So fundamental issues: - the tables are too verbose (not normalised) practicality. this is a legacy problem - relations were introduced as unordered sets to solve a particular problem, but their use has since outgrown their original conception. in 0.6 the relations are able to totally model ways, but in 0.5 they are not. there are several other non-normalities in the tables (i.e: *_tags), but ways/relations is not one of them. Even in 0.5 relations can be ordered using the type='...'. - the practical usage of limitations (only fetch what you can observe) is not exploited at all, while this is an issue for a renderer and a typical client that wants to use the data it is exploited in the map call to return only visible nodes, ways and relations. I think you mean intersect here or not? Because if you already have a call that gives back a truely right map. The only thing that should be implemented in the editor, that you cannot edit nodes on the boundaries. Now that actually is trivial (begin and endpoints of the polyline). Anyway the resolve scenario sounds like Microsoft, you cannot use character blablabla in your filename, because we say so. NTFS disallows /, as do most unix filesystems. macos x disallows : http://en.wikipedia.org/wiki/Filename#Comparison_of_file_name_limitations these limitations are usually imposed to prevent confusion. i.e: / is always the root directory, not a file called / in the current directory. There are more things that are less obvious ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Frederik Ramm wrote: Stefan de Konink wrote: The latter still requires the same client/server modifications + modifications in all current node relations. Hence it will cost more time. No it doesn't; 99.9% of all ways don't violate the new criteria so all tools will work perfectly for them. The problem is the actual search for and modification of the violators. Like the current problem with ways/relation of size 0; invalid references etc. Even in 0.5 relations can be ordered using the type='...'. You're getting (more) ridiculous. What are you trying to prove here? That you're right and the rest is wrong? That you are a masterful programmer and the others just dumb fiddlers? Matt claimed it was impossible to do order in relations within API 0.5. That argument is wrong. That is the only thing I'm proving here. And I did make an error because I should talked about role instead of type. You have meanwhile reached a point where I'd rather see the project go down in flames than implement a single one of your ideas. Do it yourself or go to hell. I am doing it myself, don't worry. And I don't get stopped by any flames on mailinglists :) Happy new year to you too :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Tue, Dec 30, 2008 at 1:05 PM, Stefan de Konink ste...@konink.de wrote: Frederik Ramm wrote: Stefan de Konink wrote: Even in 0.5 relations can be ordered using the type='...'. You're getting (more) ridiculous. What are you trying to prove here? That you're right and the rest is wrong? That you are a masterful programmer and the others just dumb fiddlers? Matt claimed it was impossible to do order in relations within API 0.5. That argument is wrong. That is the only thing I'm proving here. And I did make an error because I should talked about role instead of type. you're right - i should have said ways can't be transparently modelled using relations in the 0.5 database structure. you could use role for ordering, but the ordering would have to be imposed client-side. the client also has to deal with whatever meaning is assigned to other relation members of the way or members with non-numeric roles. its much easier for the client with ways... relations are an incredibly powerful structure, which can be used to model just about anything. whether they *should* be used is a different, mostly non-technical, discussion. for example, it is possible to build a turing complete computer with a single, incredibly powerful instruction (subleq / subneg) but that doesn't mean i want to program it :-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? If not, then it is pretty useless to have an infrastructure that is capable of holding an infinite amount of nodes and one that is not ;) IMHO ways should be restricted to more like 256 nodes, and relations should be unrestricted. I don't see how relations and ways should be similar in this respect. I can imagine a system that downloaded only part of a relationship, and had abstract add/remove object X from relationship Y methods, but with ways it seems much more complicated because of ordering. Robert (Jamie) Munro -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklaPEQACgkQz+aYVHdncI2EyQCg080k2S7tT3JmbiVIJlNYQO3v nHEAn28U0TLQr8gzK81hCUAJXsONl89y =/iR1 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Robert (Jamie) Munro wrote: Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? If not, then it is pretty useless to have an infrastructure that is capable of holding an infinite amount of nodes and one that is not ;) IMHO ways should be restricted to more like 256 nodes, and relations should be unrestricted. I don't see how relations and ways should be similar in this respect. osm version=0.5 generator=OpenStreetMap server way id=100 visible=true timestamp=2008-04-16T16:42:42+01:00 user=ck3d nd ref=260904/ nd ref=260897/ nd ref=260898/ nd ref=185986175/ nd ref=260899/ nd ref=260900/ nd ref=260901/ nd ref=260902/ nd ref=260903/ nd ref=260904/ tag k=highway v=secondary/ tag k=ref v=St 2069/ tag k=junction v=roundabout/ tag k=created_by v=JOSM/ /way /osm vs osm version=0.5 generator=OpenStreetMap server relation id=100 visible=true timestamp=2008-04-16T16:42:42+01:00 user=ck3d member type=node ref=260904 role=1 / member type=node ref=260897 role=2 / member type=node ref=260898 role=3 / member type=node ref=185986175 role=4 / member type=node ref=260899 role=5 / member type=node ref=260900 role=6 / member type=node ref=260901 role=7 / member type=node ref=260902 role=8 / member type=node ref=260903 role=9 / member type=node ref=260904 role=10 / tag k=highway v=secondary/ tag k=ref v=St 2069/ tag k=junction v=roundabout/ tag k=created_by v=JOSM/ /way /osm I do hope you understand now :) I can imagine a system that downloaded only part of a relationship, and had abstract add/remove object X from relationship Y methods, but with ways it seems much more complicated because of ordering. It is not a problem at all. Because the ordering is explicit as you can observe. A bbox request would bbox on the nodes, but not return all the nodes for the complete relational-way. The editor should simply say, you cannot extend an incomplete way or move the last point. Or in best case, offer an option to download it fully. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
2008/12/30 Robert (Jamie) Munro rjmu...@arjam.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? If not, then it is pretty useless to have an infrastructure that is capable of holding an infinite amount of nodes and one that is not ;) IMHO ways should be restricted to more like 256 nodes, and relations should be unrestricted. I don't see how relations and ways should be similar in this respect. I can imagine a system that downloaded only part of a relationship, and had abstract add/remove object X from relationship Y methods, but with ways it seems much more complicated because of ordering. Because it's the same problem. It's about ensuring that API objects remain a managable size, not just for upload, but download and editing too. The add/remove would be abandoning our REST API interface which isn't exactly a trivial change for anybody, plus 0.6 makes relations ordered anyway, so it'd be just as complicated as doing it for ways. Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Tue, Dec 30, 2008 at 4:20 PM, Robert (Jamie) Munro rjmu...@arjam.net wrote: IMHO ways should be restricted to more like 256 nodes Oh god I hope not. Coastlines by themselves are tens of millions of nodes and there's already a huge number of ways needed to do it. Reducing the number of nodes in a way to 256 is a useless waste of time. Sometimes you really do need to store lots of nodes. Eventually country boundaries are going to be the same order of magnitude and I don't want to see madness like having you split an island boundary in two just because it's 257 nodes. 2000 I think is in the right range, bigger would be nice but no smaller please. Have a nice day, -- Martijn van Oosterhout klep...@gmail.com http://svana.org/kleptog/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Martijn van Oosterhout wrote: On Tue, Dec 30, 2008 at 4:20 PM, Robert (Jamie) Munro rjmu...@arjam.net wrote: IMHO ways should be restricted to more like 256 nodes Oh god I hope not. Coastlines by themselves are tens of millions of nodes and there's already a huge number of ways needed to do it. Reducing the number of nodes in a way to 256 is a useless waste of time. Sometimes you really do need to store lots of nodes. Not wanting to plead for a certain limit of nodes per way, but what would be the technical problem with a single entity consisting of a million 2-node ways, as opposed to a way of a million nodes? Ok, it will take some time to connect those million nodes, but at present, osmarender likes the million ways better than the million-node way. (ok, have not tested a million ways, the examples where osmarender barfs are usually in the tens of thousands of nodes). Eventually country boundaries are going to be the same order of magnitude and I don't want to see madness like having you split an island boundary in two just because it's 257 nodes. I thought that for ease of rendering (be it osmarender, or JOSM, or maybe Merkaartor) it is always better to let the number of nodes per way not grow out of bounds. Maarten ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Maarten Deen wrote: Ok, it will take some time to connect those million nodes, but at present, osmarender likes the million ways better than the million-node way. But think *why* this is; I presume osmarender is fed the million nodes because it intersects that way. There are situations where this can happen in real life even when partial ways are implemented, for example zooming out to world view. This is a scenario your renderer must solve; because the renderer knows its precision for that render, not the data provider. A typical editor is already limited in what it can 'fetch'. In my previous mails you can find the other arguments that make more sense as resolution. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Tue, Dec 30, 2008 at 10:18 PM, Maarten Deen md...@xs4all.nl wrote: Not wanting to plead for a certain limit of nodes per way, but what would be the technical problem with a single entity consisting of a million 2-node ways, as opposed to a way of a million nodes? maybe we could call these 2-node ways segments? ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Matt Amos wrote: On Tue, Dec 30, 2008 at 10:18 PM, Maarten Deen md...@xs4all.nl wrote: Not wanting to plead for a certain limit of nodes per way, but what would be the technical problem with a single entity consisting of a million 2-node ways, as opposed to a way of a million nodes? maybe we could call these 2-node ways segments? ;-) Maybe he made a typo and ment 2k-node... but that is just me parsing the meaning behind a message. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Wed, Dec 31, 2008 at 3:30 AM, Stefan de Konink ste...@konink.de wrote: Maybe he made a typo and ment 2k-node... but that is just me parsing the meaning behind a message. other than the obvious fact that 10^6 * 2k has three orders of magnitude more nodes* than a 10^6 node way? cheers, matt * after merging, clearly. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Matt Amos wrote: On Wed, Dec 31, 2008 at 3:30 AM, Stefan de Konink ste...@konink.de wrote: Maybe he made a typo and ment 2k-node... but that is just me parsing the meaning behind a message. other than the obvious fact that 10^6 * 2k has three orders of magnitude more nodes* than a 10^6 node way? The question seems to be is there really any better performance when you use a million 2k node ways opposed to a million node way (that is obviously not 2k). Both are big. Will performance it the 2k limited way worse than the way that is 'all in one'. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink wrote: Matt Amos wrote: On Tue, Dec 30, 2008 at 10:18 PM, Maarten Deen md...@xs4all.nl wrote: Not wanting to plead for a certain limit of nodes per way, but what would be the technical problem with a single entity consisting of a million 2-node ways, as opposed to a way of a million nodes? maybe we could call these 2-node ways segments? ;-) That would be... so 0.4 :-P Maybe he made a typo and ment 2k-node... but that is just me parsing the meaning behind a message. No, I meant 2, to give the contrast between two ways of equal length but different datatechnical makeup. Obviously there are drawbacks to that, and I'm absolutely not suggesting that such ways should be created, but currently the other option can also give problems. So don't go to extremes, either way. Where the solution for rendering is relatively easy (drop everything outside your viewingbox), for editors it is not. You will have to keep the whole way for the off chance that you zoom out in the editor or really want to edit the whole way (like splitting it). Regards, Maarten ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink ste...@konink.de writes: Matthias Julius wrote: ... but not a tree of relations. Every relation is a tree (almost). So I don't see a particular problem in making a boundary a relation, too. Some people wanted to limit ways/relations to 2k of nodes. If you do allow relations to have all these extra subrelations then what are you going to solve in the end? This is enforcing a clueless limit and not solving the fundamental problem why this limit is justified; and no that has nothing to do with data storage. Not subrelations, but a relation with 2000 ways with 2000 nodes each. And no it has nothing to do with storage, only with data managable for users. As I mentioned earlier, there is no need for even the concept 'way'; since you can store a relation with tag highway=whatever. So fundamental issues: I think keeping ways as linear primitives is a good idea. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Matthias Julius wrote: Stefan de Konink ste...@konink.de writes: Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? A relation with 2000 ways with 2000 nodes each can be an area with almost 4,000,000 nodes (3,998,000 to be exact). That will get you pretty far. Hence you are making a tree... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
What is the TRAPI server? (besides apparently very quick?) Any specs/munin graphs for the curious? It's pretty much handling the load all by itself today. I operate the physical server, not the application itself. The server specs are: Quad Xeon 2.13GHz, 8GB RAM, 2x1TB SATA DISK Stats are available here, for now: http://80.167.103.102/html/munin/openhost.dk/host1.openhost.dk.html Questions about TRAPI should be directed at Blars Blarson. Best Regards, - Johnny Carlsen ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink ste...@konink.de writes: Matthias Julius wrote: Stefan de Konink ste...@konink.de writes: Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? A relation with 2000 ways with 2000 nodes each can be an area with almost 4,000,000 nodes (3,998,000 to be exact). That will get you pretty far. Hence you are making a tree... ... but not a tree of relations. Every relation is a tree (almost). So I don't see a particular problem in making a boundary a relation, too. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Matthias Julius wrote: ... but not a tree of relations. Every relation is a tree (almost). So I don't see a particular problem in making a boundary a relation, too. Some people wanted to limit ways/relations to 2k of nodes. If you do allow relations to have all these extra subrelations then what are you going to solve in the end? This is enforcing a clueless limit and not solving the fundamental problem why this limit is justified; and no that has nothing to do with data storage. As I mentioned earlier, there is no need for even the concept 'way'; since you can store a relation with tag highway=whatever. So fundamental issues: - the tables are too verbose (not normalised) - the tables imply limits that are not required from a database standpoint - the practical usage of limitations (only fetch what you can observe) is not exploited at all, while this is an issue for a renderer and a typical client that wants to use the data Anyway the resolve scenario sounds like Microsoft, you cannot use character blablabla in your filename, because we say so. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Mon, Dec 29, 2008 at 10:26 PM, Stefan de Konink ste...@konink.de wrote: Some people wanted to limit ways/relations to 2k of nodes. If you do allow relations to have all these extra subrelations then what are you going to solve in the end? This is enforcing a clueless limit and not solving the fundamental problem why this limit is justified; and no that has nothing to do with data storage. practicality. since we all agree that no-one wants to download a 100k+ node way, there are two immediately obvious solutions: change the API to return partial ways or disallow ways longer than some arbitrary limit. the former requires many changes on the server and client, the latter requires many fewer. the former might be a better solution (fsov better) and we would welcome patches to the server, josm, potlatch and merkaartor which implement it. As I mentioned earlier, there is no need for even the concept 'way'; since you can store a relation with tag highway=whatever. So fundamental issues: - the tables are too verbose (not normalised) practicality. this is a legacy problem - relations were introduced as unordered sets to solve a particular problem, but their use has since outgrown their original conception. in 0.6 the relations are able to totally model ways, but in 0.5 they are not. there are several other non-normalities in the tables (i.e: *_tags), but ways/relations is not one of them. this may change (or be fixed, depending on your point of view) in future API versions. as shaun said: one step at a time. - the tables imply limits that are not required from a database standpoint the API certainly does, but this is related to the point above. just because there is no technical reason from a database standpoint doesn't mean there isn't a reason. - the practical usage of limitations (only fetch what you can observe) is not exploited at all, while this is an issue for a renderer and a typical client that wants to use the data it is exploited in the map call to return only visible nodes, ways and relations. Anyway the resolve scenario sounds like Microsoft, you cannot use character blablabla in your filename, because we say so. NTFS disallows /, as do most unix filesystems. macos x disallows : http://en.wikipedia.org/wiki/Filename#Comparison_of_file_name_limitations these limitations are usually imposed to prevent confusion. i.e: / is always the root directory, not a file called / in the current directory. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] ROMA servers down - osmosis large way problem
Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote: There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
It's not the ROMA script that's failing - osmosis is failing while it's trying to update the database using changesets from the main API. Whatever db schema the main API uses must allow for this, while the one for osmosis that was used to setup at least my server does not. -Jeremy Original Message --- Jeremy Adams wrote: There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote:
It's not the ROMA script that's failing - osmosis is failing while
it's trying to update the database using changesets from the main
API.
Whatever db schema the main API uses must allow for this, while the
one for osmosis that was used to setup at least my server does not.
Ok; I cannot understand the difficulty in it :{ I mean it is just the
parsing of a subtree.
My own update mechanism is rather untested. Otherwise I would suggest
that. It basically allows any user to submit changesets as part of their
edits. If my relative resolver is done I'll propose it as alternative
API mechanism that doesn't care about changesets anymore because it is a
changeset by itself.
(people that are interested in it know where to look)
Stefan
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
2008/12/29 Jeremy Adams mile...@king-nerd.com Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy The same problem happened on 31st October... Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple shema / smallint out of range On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote: Hi, i just discovered that osmosis was not able to apply the hourly osc file starting 2008-10-29T20:00:00Z - It failed with: 2008-10-31 11:09:52 CET ERROR: smallint out of range 2008-10-31 11:09:52 CET STATEMENT: INSERT INTO way_nodes (way_id, node_id, sequence_id) VALUES ($1, $2, $3) Is there a way with 2^16 aka 65536 nodes? Or did someone manage to enter a completely broken sequence number? ... [show rest of quotehttp://www.nabble.com/psql-osmosis-simple-shemasmallint-out-of-range-td20263104.html# ] ... [rest of quote removed] Its 2^15 because it signed - and yes - somebody managed to get abovE: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... d ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
+--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams mile...@king-nerd.com wrote: | It's not the ROMA script that's failing - osmosis is failing while it's | trying to update the database using changesets from the main API. | | Whatever db schema the main API uses must allow for this, while the one | for osmosis that was used to setup at least my server does not. I don't really see why it would be a db schema problem (it didn't fail on my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3 fields :-) Also, I've added the TRAPI server to the ROMA load balancer because well, there was only me for a few hours, and it was making me nervous, which was a good thing, because my ROMA is down now. (A drive earlier today failed, I'll try to fix it in a week when I come back from holiday.) -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... Both of my databases broke down again as i dropped the databases and recreated them - right now converting again ... I have no clue if this solves it - forgot already what i did last time ;) ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer; Hopefully postgres is clever enough to take over the old content and recreate the index ... Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
+--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. Ok fundamental problem: premature optimisations. How much disk space have been saved with this optimisation until now? (With an alter table you would be able to see it.) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Weird - I checked the schema that I used and it's a smallint. That must have been changed recently. I've altered the column type on my server - we'll see what happens when it's finished. -Jeremy Original Message --- +--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 20:56, Mathieu Arnold wrote: +--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); I think this is a problem that was fixed the last time, though the people running the ROMA servers haven't updated to the latest svn version of osmosis to get the latest updates. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 20:50, Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. From API 0.6 there is a limit of 2,000 nodes in a way. http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225 and http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9 Hopefully someone will fix the way prior to the new api transition. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? If not, then it is pretty useless to have an infrastructure that is capable of holding an infinite amount of nodes and one that is not ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Sun, Dec 28, 2008 at 09:15:34PM +0100, Stefan de Konink wrote: Subject: Re: [OSM-dev] ROMA servers down - osmosis large way problem Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. ...and I hope this limit will be dropped again dropped and allow partial modification for ways opposed to uploading the entire way. Would be kind of complicated i guess. The point is that you have number 1..100 on a way - you edit the range from 50-60 and insert 5 nodes than you'll have 50-65 which will have an overlap with the ones in the database (sequence wise) so the database/api needs to reorganize sequence numbers of nodes you didnt edit. Id say - very ugly .. Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Florian Lohoff wrote: Would be kind of complicated i guess. The point is that you have number 1..100 on a way - you edit the range from 50-60 and insert 5 nodes than you'll have 50-65 which will have an overlap with the ones in the database (sequence wise) so the database/api needs to reorganize sequence numbers of nodes you didnt edit. Id say - very ugly .. You would get the current version of that way back when you download. A subtree can be seen as ordered; for sake of redundancy we will add the sequence of moment of serving. Using the locking assumption we could just add remove the ways that where added or removed from the relation by the fantastic model that Microsoft Research has thought out (it is not a joke) id.subid. nds id=1.1 ... to insert a node after the first node. nds id=1.-1 ... to insert a node before the first node. Now this would work out pretty well, aka server verifies that the current changeset equals the submitted changes and inserts the nodes, and reorders the index. But now what if we do have transactions; - Server based best effort, like diff does with fuzzy matching - Just tell the user you are too late this is the way now, and let your editor of choice fix the mess. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. ...and I hope this limit will be dropped again dropped It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Frederik Ramm wrote: Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink wrote: So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. POST changeset/#id/uploadUploads a diff into a changeset transactionally. Returns XML with the committed versions and IDs of the diff elements. Coded, Tested Looking at the specs that is already implemented. .. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 22:10, Stefan de Konink wrote: Frederik Ramm wrote: Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) That is not what relations are for. You determine which country is in based on the latitude and longitude of the nodes of the items, and a bounding polygon for the country that you are interested in. It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. One step at a time. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) Someone who is so quick with taking complex programming for granted (... simply implement partial editing of ways...) will surely not be foiled by the requirement to access a second level of relations which seems rather trivial in comparison. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Thank you but we need programmers, not advice. Sending a complete changeset within one request is possible BTW. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) That is not what relations are for. You determine which country is in based on the latitude and longitude of the nodes of the items, and a bounding polygon for the country that you are interested in. That polygon just happens to be the cutted way/relation, or whatever category you want to assign to it. Sence it is not one thing anymore, it is not cyclic, thus broken thus mathematically extremely inefficient to calculate upon. It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. One step at a time. (As mentioned; it is already in there...) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) Someone who is so quick with taking complex programming for granted (... simply implement partial editing of ways...) will surely not be foiled by the requirement to access a second level of relations which seems rather trivial in comparison. I do what everyone does that is smart, not reinvent the wheel. Geos is for us geometry-dropouts. There is no a priori need to break things up. If you do want to break things up, implement server/clientside what I suggested before, partial ways (as in return less nodes) limited by bboxing. That has a effect on any form of processing. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Thank you but we need programmers, not advice. Sending a complete changeset within one request is possible BTW. You have already implemented it; (I have too btw). Since I'm not going to code Ruby or Java 'for fun', I'll stick with the C implementation of the API. Like other people work on other editors not in Java. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
What is the TRAPI server? (besides apparently very quick?) Any specs/munin graphs for the curious? It's pretty much handling the load all by itself today. -Jeremy Original Message --- +--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams mile...@king-nerd.com wrote: | It's not the ROMA script that's failing - osmosis is failing while it's | trying to update the database using changesets from the main API. | | Whatever db schema the main API uses must allow for this, while the one | for osmosis that was used to setup at least my server does not. I don't really see why it would be a db schema problem (it didn't fail on my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3 fields :-) Also, I've added the TRAPI server to the ROMA load balancer because well, there was only me for a few hours, and it was making me nervous, which was a good thing, because my ROMA is down now. (A drive earlier today failed, I'll try to fix it in a week when I come back from holiday.) -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Seems like you're all on top of the problem now. I updated the schema creation script on 3rd November to use an int instead of smallint. Internally osmosis doesn't care, it uses a 32 bit integer (more accurately, it uses a list index which is 32 bit). It is just the schema itself which was using a smallint to minimise disk usage. It should be possible to change the column type without losing data. If there's anything I can do let me know. Note that the 0.6 version of the schema still uses a smallint, I possibly should change that in case people start testing it with data migrated from a 0.5 planet. D Tucny wrote: 2008/12/29 Jeremy Adams mile...@king-nerd.com mailto:mile...@king-nerd.com Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy The same problem happened on 31st October... Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple shema / smallint out of range On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote: Hi, i just discovered that osmosis was not able to apply the hourly osc file starting 2008-10-29T20:00:00Z - It failed with: 2008-10-31 11:09:52 CET ERROR: smallint out of range 2008-10-31 11:09:52 CET STATEMENT: INSERT INTO way_nodes (way_id, node_id, sequence_id) VALUES ($1, $2, $3) Is there a way with 2^16 aka 65536 nodes? Or did someone manage to enter a completely broken sequence number? ... [show rest of quote http://www.nabble.com/psql-osmosis-simple-shemasmallint-out-of-range-td20263104.html#] ... [rest of quote removed] Its 2^15 because it signed - and yes - somebody managed to get abovE: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... d ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. -Jeremy Original Message --- On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... Both of my databases broke down again as i dropped the databases and recreated them - right now converting again ... I have no clue if this solves it - forgot already what i did last time ;) ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer; Hopefully postgres is clever enough to take over the old content and recreate the index ... Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote: I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. It should not do anything on your CPU; such actions are disk operations. Depending on the DBS it is possible that it is faster to copy a column and merge it back later than to alter. Does your column have constraints on it? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: On 28 Dec 2008, at 20:50, Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. From API 0.6 there is a limit of 2,000 nodes in a way. http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225 and http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9 Hopefully someone will fix the way prior to the new api transition. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev There are actually a number of ways that are excessively too large currently, many of the largest are geopolitical boundaries from the Geobase import (the import was 162,000 nodes in around 500 ways). We've been discussing how to reduce them into more manageable chunks over on talk-ca. So they will all be under the limit by the time the transition happens. -Jason Reid ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink ste...@konink.de writes: Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? A relation with 2000 ways with 2000 nodes each can be an area with almost 4,000,000 nodes (3,998,000 to be exact). That will get you pretty far. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
