Re: [OSM-dev] which Java Verison to use (1.6 vs. 1.5); was: Error compiling osmosis
On 28 Dec 2008, at 00:36, Brett Henderson wrote: Java 1.5 is more widely available than Java 1.6. OSX in particular hasn't had support for 1.6, not sure if that's changed yet. The current status of Java 1.6 on Mac OS X is that Apple have decided to only support it on the Intel 64 bit platform. PPC and Intel 32 bit have been left in the cold by Apple. There is also a bug that Apple have acknowledged, that means that ant will not find Java 1.6, using the default mechanisms supplied by Apple, where Java 1.6 can be installed. Hopefully the next version of the developer tools will have this fixed. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] which Java Verison to use (1.6 vs. 1.5); was: Error compiling osmosis
It is OK that Osmosis supports just 1.6, because it has no UI (has it?). So there are 2 reasons: - You use it more on servers with Linux, so 1.6 is supported. - The user has the competence to deal with this kind of things, including setting up his own DB. JOSM - in opposite - should be as easy as possible to use (= downloading and starting to edit), so that as many people as possible contribute to OSM. So focus on new features and don't try to support ancient versions. Best, Andi Brett Henderson schrieb: Sending this to dev because I'm curious to hear thoughts from the JOSM guys. I agree it would be nice to standardise but the reasons for the difference as I understand it are: JOSM is an end user tool where wide platform support is necessary. Java 1.5 is more widely available than Java 1.6. OSX in particular hasn't had support for 1.6, not sure if that's changed yet. JOSM is maintaining compatibility with Java 1.5. *However* it should be possible to compile JOSM on Java 1.6, it just won't run on 1.5 if you do so. There may be some warnings around the use of the @Override annotation (I don't know the details here) but I don't think these should be show stoppers. Osmosis is newer than JOSM, and is less end-user focused. I don't support 1.5 because I'm using some newer features of the 1.6 platform. From memory, these are Java 2D libraries required for accurate polygon support, concurrent libraries, and collection libraries. Of these it might be possible to support 1.5 with some additional effort but the polygon support in particular is hard to do properly on 1.5 because 1.6 added double accuracy 2D calculations. Here's my suggestion. OpenJDK is a Java 1.6 platform. I assume that is what Debian will provide out of the box. Compile both JOSM and Osmosis using this 1.6 platform. They will both then run on the OpenJDK provided by Debian. Neither will run on older 1.5 JDKs. You aren't building binary distributions of JOSM for cross-platform use therefore you don't need to support 1.5. 1.6 provides some very useful features, it has been released for around 2 years now, and is supported on the vast majority of platforms out there. Supporting 1.5 just isn't a high priority for me. With the open source OpenJDK out there I was hoping the need to support ancient java releases would be eliminated. Thoughts welcome. Brett Joerg Ostertag (OSM Tettnang/Germany) wrote: OK ... so josm doesn't want java 1.6 and osmosis doesn't want java 1.5 This doesn't really make it easy to create debian packages out of these ... Couldn't we agree on one Version of Java for all OSM Software. This would macke packaging much easier for me. - Joerg On Mittwoch 24 Dezember 2008, you wrote: The minimum required java version for osmosis is 1.6. You appear to be using version 1.5. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Your whishlist for Debian Packages?
Hi, as most of you probably know I'm providing debian packages for the most common tools needed to work with OpenStreetMap. What I'd like to know is: - which additional tools from the osm-svn would you like to see added to the debian repository? - which other debian based platforms would you like to see suported? examples would be: debian-etch, Ubuntu-xx, ... If you give me a preferences list, I can start with the most wanted ones More description on the existing debian repository can be found at: http://www.gpsdrive.de/development/debian.shtml -- Jörg (Germany, Tettnang) http://www.ostertag.name/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What about moving wmsplugin/webkit-image to a pplication/utils? Was: Need help compiling wmsplugin /webkit-image.cpp on Debian-lenny
On Sonntag 28 Dezember 2008, Joerg Ostertag (OSM Tettnang/Germany) wrote: On Donnerstag 25 Dezember 2008, Simon Ward wrote: On Thu, Dec 25, 2008 at 08:32:04AM +0100, Joerg Ostertag (OSM Tettnang/Germany) wrote: Hi, I need help compiling wmsplugin/webkit-image.cpp on my Debian-lenny machine. when I try to compile I get the following errors: ... It works :-) I changed the Makefile and now I compiled the package with the new Makefile. For now I compiled it into the openstreetmap-josm package. What about moving the webkit-image stuff into the utils tree? This way I would compile the architecture dependent things in the debian-package openstreetmap-utils and there would not be any need to split the openstreetmap-josm package into different packages for different cpu-types. -- Jörg (Germany, Tettnang) http://www.ostertag.name/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] ROMA servers down - osmosis large way problem
Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote: There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
It's not the ROMA script that's failing - osmosis is failing while it's trying to update the database using changesets from the main API. Whatever db schema the main API uses must allow for this, while the one for osmosis that was used to setup at least my server does not. -Jeremy Original Message --- Jeremy Adams wrote: There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote:
It's not the ROMA script that's failing - osmosis is failing while
it's trying to update the database using changesets from the main
API.
Whatever db schema the main API uses must allow for this, while the
one for osmosis that was used to setup at least my server does not.
Ok; I cannot understand the difficulty in it :{ I mean it is just the
parsing of a subtree.
My own update mechanism is rather untested. Otherwise I would suggest
that. It basically allows any user to submit changesets as part of their
edits. If my relative resolver is done I'll propose it as alternative
API mechanism that doesn't care about changesets anymore because it is a
changeset by itself.
(people that are interested in it know where to look)
Stefan
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
2008/12/29 Jeremy Adams mile...@king-nerd.com Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy The same problem happened on 31st October... Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple shema / smallint out of range On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote: Hi, i just discovered that osmosis was not able to apply the hourly osc file starting 2008-10-29T20:00:00Z - It failed with: 2008-10-31 11:09:52 CET ERROR: smallint out of range 2008-10-31 11:09:52 CET STATEMENT: INSERT INTO way_nodes (way_id, node_id, sequence_id) VALUES ($1, $2, $3) Is there a way with 2^16 aka 65536 nodes? Or did someone manage to enter a completely broken sequence number? ... [show rest of quotehttp://www.nabble.com/psql-osmosis-simple-shemasmallint-out-of-range-td20263104.html# ] ... [rest of quote removed] Its 2^15 because it signed - and yes - somebody managed to get abovE: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... d ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
+--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams mile...@king-nerd.com wrote: | It's not the ROMA script that's failing - osmosis is failing while it's | trying to update the database using changesets from the main API. | | Whatever db schema the main API uses must allow for this, while the one | for osmosis that was used to setup at least my server does not. I don't really see why it would be a db schema problem (it didn't fail on my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3 fields :-) Also, I've added the TRAPI server to the ROMA load balancer because well, there was only me for a few hours, and it was making me nervous, which was a good thing, because my ROMA is down now. (A drive earlier today failed, I'll try to fix it in a week when I come back from holiday.) -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... Both of my databases broke down again as i dropped the databases and recreated them - right now converting again ... I have no clue if this solves it - forgot already what i did last time ;) ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer; Hopefully postgres is clever enough to take over the old content and recreate the index ... Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
+--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. Ok fundamental problem: premature optimisations. How much disk space have been saved with this optimisation until now? (With an alter table you would be able to see it.) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Weird - I checked the schema that I used and it's a smallint. That must have been changed recently. I've altered the column type on my server - we'll see what happens when it's finished. -Jeremy Original Message --- +--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 20:56, Mathieu Arnold wrote: +--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff f...@rfc822.org wrote: | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: | So what is the fundamental problem that your tools break on 40k of | nodes? Bbox them on the request, and return a partial result that will | work just as good for rendering. | | The db schema uses smallint for the node number on the way - so you may | only address -32768 to +32767 number of nodes on a way - 40k nodes | is out of scope of a smallint so osmosis breaks on importing the | changeset. I'm not certain what we're talking about, but in the version of osmosis I found, the pgsql_simple_schema_0.5.sql file contains : CREATE TABLE way_nodes ( way_id bigint NOT NULL, node_id bigint NOT NULL, sequence_id int NOT NULL ); I think this is a problem that was fixed the last time, though the people running the ROMA servers haven't updated to the latest svn version of osmosis to get the latest updates. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 20:50, Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. From API 0.6 there is a limit of 2,000 nodes in a way. http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225 and http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9 Hopefully someone will fix the way prior to the new api transition. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? If not, then it is pretty useless to have an infrastructure that is capable of holding an infinite amount of nodes and one that is not ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On Sun, Dec 28, 2008 at 09:15:34PM +0100, Stefan de Konink wrote: Subject: Re: [OSM-dev] ROMA servers down - osmosis large way problem Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. ...and I hope this limit will be dropped again dropped and allow partial modification for ways opposed to uploading the entire way. Would be kind of complicated i guess. The point is that you have number 1..100 on a way - you edit the range from 50-60 and insert 5 nodes than you'll have 50-65 which will have an overlap with the ones in the database (sequence wise) so the database/api needs to reorganize sequence numbers of nodes you didnt edit. Id say - very ugly .. Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Florian Lohoff wrote: Would be kind of complicated i guess. The point is that you have number 1..100 on a way - you edit the range from 50-60 and insert 5 nodes than you'll have 50-65 which will have an overlap with the ones in the database (sequence wise) so the database/api needs to reorganize sequence numbers of nodes you didnt edit. Id say - very ugly .. You would get the current version of that way back when you download. A subtree can be seen as ordered; for sake of redundancy we will add the sequence of moment of serving. Using the locking assumption we could just add remove the ways that where added or removed from the relation by the fantastic model that Microsoft Research has thought out (it is not a joke) id.subid. nds id=1.1 ... to insert a node after the first node. nds id=1.-1 ... to insert a node before the first node. Now this would work out pretty well, aka server verifies that the current changeset equals the submitted changes and inserts the nodes, and reorders the index. But now what if we do have transactions; - Server based best effort, like diff does with fuzzy matching - Just tell the user you are too late this is the way now, and let your editor of choice fix the mess. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. ...and I hope this limit will be dropped again dropped It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Frederik Ramm wrote: Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink wrote: So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. POST changeset/#id/uploadUploads a diff into a changeset transactionally. Returns XML with the committed versions and IDs of the diff elements. Coded, Tested Looking at the specs that is already implemented. .. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
On 28 Dec 2008, at 22:10, Stefan de Konink wrote: Frederik Ramm wrote: Stefan de Konink wrote: Shaun McDonald wrote: From API 0.6 there is a limit of 2,000 nodes in a way. Btw; Do you impose this also on a relation? Yes. So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) That is not what relations are for. You determine which country is in based on the latitude and longitude of the nodes of the items, and a bounding polygon for the country that you are interested in. It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. One step at a time. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Hi, Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) Someone who is so quick with taking complex programming for granted (... simply implement partial editing of ways...) will surely not be foiled by the requirement to access a second level of relations which seems rather trivial in comparison. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Thank you but we need programmers, not advice. Sending a complete changeset within one request is possible BTW. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: So you actually suggest to implement a tree of relations to render a country? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) That is not what relations are for. You determine which country is in based on the latitude and longitude of the nodes of the items, and a bounding polygon for the country that you are interested in. That polygon just happens to be the cutted way/relation, or whatever category you want to assign to it. Sence it is not one thing anymore, it is not cyclic, thus broken thus mathematically extremely inefficient to calculate upon. It won't. Partial modification of ways is a good idea but we lack the coders to implement it in API and clients. It might be done at a later time but surely not with 0.6. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. One step at a time. (As mentioned; it is already in there...) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? Totally break lookups such as 'is X in country Y', you will make friends that is for sure :) Someone who is so quick with taking complex programming for granted (... simply implement partial editing of ways...) will surely not be foiled by the requirement to access a second level of relations which seems rather trivial in comparison. I do what everyone does that is smart, not reinvent the wheel. Geos is for us geometry-dropouts. There is no a priori need to break things up. If you do want to break things up, implement server/clientside what I suggested before, partial ways (as in return less nodes) limited by bboxing. That has a effect on any form of processing. So you really want people first to change everything to 0.6 and then to something else ;) I wanted to write longer about this, but I think allowing everyone to send just a complete changeset (within one request) ditch REST for updating will give much more momentum, and requires less code. Thank you but we need programmers, not advice. Sending a complete changeset within one request is possible BTW. You have already implemented it; (I have too btw). Since I'm not going to code Ruby or Java 'for fun', I'll stick with the C implementation of the API. Like other people work on other editors not in Java. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
What is the TRAPI server? (besides apparently very quick?) Any specs/munin graphs for the curious? It's pretty much handling the load all by itself today. -Jeremy Original Message --- +--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams mile...@king-nerd.com wrote: | It's not the ROMA script that's failing - osmosis is failing while it's | trying to update the database using changesets from the main API. | | Whatever db schema the main API uses must allow for this, while the one | for osmosis that was used to setup at least my server does not. I don't really see why it would be a db schema problem (it didn't fail on my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3 fields :-) Also, I've added the TRAPI server to the ROMA load balancer because well, there was only me for a few hours, and it was making me nervous, which was a good thing, because my ROMA is down now. (A drive earlier today failed, I'll try to fix it in a week when I come back from holiday.) -- Mathieu Arnold ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Seems like you're all on top of the problem now. I updated the schema creation script on 3rd November to use an int instead of smallint. Internally osmosis doesn't care, it uses a 32 bit integer (more accurately, it uses a list index which is 32 bit). It is just the schema itself which was using a smallint to minimise disk usage. It should be possible to change the column type without losing data. If there's anything I can do let me know. Note that the 0.6 version of the schema still uses a smallint, I possibly should change that in case people start testing it with data migrated from a 0.5 planet. D Tucny wrote: 2008/12/29 Jeremy Adams mile...@king-nerd.com mailto:mile...@king-nerd.com Hey all, There's an issue with the minute change file 200812290912-200812290913.osc.gz. It contains a way which has over 40k nodes. It's an administrative boundary which appears to enclose all of Quebec. It's way id is 29309772. When osmosis attempts to process this file it fails. The relevant parts of the error appear to be Unable to insert new way node and a pgsql error ERROR: smallint out of range. I'm sure I can find the way and split it into smaller chunks, but what's the process for getting the ROMA servers running again? Can we just remove the reference to the way in the changeset and continue updating? The way should be added back in once it's split and reuploaded, correct? -Jeremy The same problem happened on 31st October... Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple shema / smallint out of range On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote: Hi, i just discovered that osmosis was not able to apply the hourly osc file starting 2008-10-29T20:00:00Z - It failed with: 2008-10-31 11:09:52 CET ERROR: smallint out of range 2008-10-31 11:09:52 CET STATEMENT: INSERT INTO way_nodes (way_id, node_id, sequence_id) VALUES ($1, $2, $3) Is there a way with 2^16 aka 65536 nodes? Or did someone manage to enter a completely broken sequence number? ... [show rest of quote http://www.nabble.com/psql-osmosis-simple-shemasmallint-out-of-range-td20263104.html#] ... [rest of quote removed] Its 2^15 because it signed - and yes - somebody managed to get abovE: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... d ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. -Jeremy Original Message --- On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote: osm= select max(sequence_id) from way_nodes; max --- 39767 (1 row) osm= select * from way_nodes where sequence_id = 39767; way_id | node_id | sequence_id --+---+- 28098452 | 308532457 | 39767 I converted the smallint to int ... Flo So, perhaps following Flo's lead and changing the smallint to int would be the best approach for dealing with getting the ROMA servers back up and running... Both of my databases broke down again as i dropped the databases and recreated them - right now converting again ... I have no clue if this solves it - forgot already what i did last time ;) ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer; Hopefully postgres is clever enough to take over the old content and recreate the index ... Flo -- Florian Lohoff f...@rfc822.org +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Jeremy Adams wrote: I ran the same command on my box about 4 hours ago and it's still cranking away. Should it really take that long to change a column type? It's got one of the CPU cores pegged, so I assume it's still doing something. It should not do anything on your CPU; such actions are disk operations. Depending on the DBS it is possible that it is faster to copy a column and merge it back later than to alter. Does your column have constraints on it? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Shaun McDonald wrote: On 28 Dec 2008, at 20:50, Florian Lohoff wrote: On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote: So what is the fundamental problem that your tools break on 40k of nodes? Bbox them on the request, and return a partial result that will work just as good for rendering. The db schema uses smallint for the node number on the way - so you may only address -32768 to +32767 number of nodes on a way - 40k nodes is out of scope of a smallint so osmosis breaks on importing the changeset. From API 0.6 there is a limit of 2,000 nodes in a way. http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225 and http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9 Hopefully someone will fix the way prior to the new api transition. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev There are actually a number of ways that are excessively too large currently, many of the largest are geopolitical boundaries from the Geobase import (the import was 162,000 nodes in around 500 ways). We've been discussing how to reduce them into more manageable chunks over on talk-ca. So they will all be under the limit by the time the transition happens. -Jason Reid ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ROMA servers down - osmosis large way problem
Stefan de Konink ste...@konink.de writes: Frederik Ramm wrote: Stefan de Konink wrote: So you actually suggest to implement a tree of relations to render a country? I'm not convinced a tree is needed. What do you have in mind to use as area that is bounded by a polygon that has more than 2k of nodes? A relation with 2000 ways with 2000 nodes each can be an area with almost 4,000,000 nodes (3,998,000 to be exact). That will get you pretty far. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Tasks for all of you
On Mon, 2008-12-22 at 17:44 +0100, Dirk Stöcker wrote: http://josm.openstreetmap.de/ticket/1677 plugin-preferences selection does not survive Update-clicking http://josm.openstreetmap.de/ticket/1442 Extrude of horizontal way only downwards, not upwards I've attached patches for these to the bugs. Matthew W. S. Bell ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Tasks for all of you
On Sun, 28 Dec 2008, Matthew W. S. Bell wrote: http://josm.openstreetmap.de/ticket/1677 plugin-preferences selection does not survive Update-clicking http://josm.openstreetmap.de/ticket/1442 Extrude of horizontal way only downwards, not upwards I've attached patches for these to the bugs. Thanks a lot. Applied. So 50% are done. Missing: http://josm.openstreetmap.de/ticket/1555 Downloaded GPS data is displayed in reverse http://josm.openstreetmap.de/ticket/1712 Incomplete ways are discarded without notice http://josm.openstreetmap.de/ticket/1654 Plugin dialog has non-clickable weblinks New tasks to have 6 again: http://josm.openstreetmap.de/ticket/1906 WMS-Plugin: Implement caching http://josm.openstreetmap.de/ticket/1903 Plugin-Update should test required version before installing http://josm.openstreetmap.de/ticket/1905 WMS-Plugin: Red error tiles do no automatically disappear and cannot be reloaded Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] Mappaint and multipolygon relation
Hello, now I'm currently implementing support of multipolygon relations in the Mappaint-Display of JOSM. One of the major problems is how to handle defective relations. My current idea is following: - I add a new string array for the relation and possibly the way datatype. - Whenever I find defective stuff, a add the error messages to the object. - Objects with errors are shown e.g. with a star in all places, where the name is shown (selection list, ...). - The validator will be able to show the errors. - Mappaint does a best guess how to handle defective stuff. Does this sounds useful? Better suggestions? Another problem will probably be the fact, that larger polygons will become slower when drawing them. Suggestions for this? Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Tasks for all of you
Dirk Stöcker writes:
http://josm.openstreetmap.de/ticket/1712
Incomplete ways are discarded without notice
This is a minimal patch. I didn't want to make too many changes
because somebody else is going to have to read this patch before they
apply it. I don't know Java very well, so if anybody spots anything
un-Javanic, please tell me.
Potential improvements:
o convert all invocations of OsmReader.parseDataSet() to
OsmReader.parseDataSetOsm() to gain access to getParseNotes().
o Doing something more sensible with the output presentation if
there are too many warnings than can fit on the screen (or will it
switch to a scrollbox?)
o It's possible that DataSet ought to be the entity that has
parseNotes. That would make for a simpler patch, but not
necessarily more correct.
Index: src/org/openstreetmap/josm/actions/OpenFileAction.java
===
--- src/org/openstreetmap/josm/actions/OpenFileAction.java (revision 1180)
+++ src/org/openstreetmap/josm/actions/OpenFileAction.java (working copy)
@@ -74,10 +74,14 @@
private void openAsData(File file) throws SAXException, IOException,
FileNotFoundException {
String fn = file.getName();
if
(ExtensionFileFilter.filters[ExtensionFileFilter.OSM].acceptName(fn)) {
-DataSet dataSet = OsmReader.parseDataSet(new
FileInputStream(file), null, Main.pleaseWaitDlg);
+OsmReader osm = OsmReader.parseDataSetOsm(new
FileInputStream(file), null, Main.pleaseWaitDlg);
+DataSet dataSet = osm.getDs();
OsmDataLayer layer = new OsmDataLayer(dataSet, file.getName(),
file);
Main.main.addLayer(layer);
layer.fireDataChange();
+if (osm.getParseNotes().length() != 0) {
+JOptionPane.showMessageDialog(Main.parent,
osm.getParseNotes());
+}
}
else
JOptionPane.showMessageDialog(Main.parent, fn+: +tr(Unknown
file extension: {0}, fn.substring(file.getName().lastIndexOf('.')+1)));
Index: src/org/openstreetmap/josm/io/OsmReader.java
===
--- src/org/openstreetmap/josm/io/OsmReader.java(revision 1180)
+++ src/org/openstreetmap/josm/io/OsmReader.java(working copy)
@@ -59,8 +59,18 @@
* The dataset to add parsed objects to.
*/
private DataSet ds = new DataSet();
+ public DataSet getDs() { return ds; }
/**
+ * Record warnings. If there were any data inconsistencies, append
+ * a newline-terminated string.
+ */
+ private String parseNotes = new String();
+ public String getParseNotes() {
+ return parseNotes;
+ }
+
+ /**
* The visitor to use to add the data to the set.
*/
private AddVisitor adder = new AddVisitor(ds);
@@ -332,6 +342,7 @@
for (long id : e.getValue()) {
Node n = findNode(id);
if (n == null) {
+ parseNotes += Skipping a way because it includes a
node that doesn't exist: + id + \n;
failed = true;
break;
}
@@ -434,6 +445,10 @@
* elemet found there is returned.
*/
public static DataSet parseDataSet(InputStream source, DataSet ref,
PleaseWaitDialog pleaseWaitDlg) throws SAXException, IOException {
+ return parseDataSetOsm(source, ref, pleaseWaitDlg).ds;
+ }
+
+ public static OsmReader parseDataSetOsm(InputStream source, DataSet ref,
PleaseWaitDialog pleaseWaitDlg) throws SAXException, IOException {
OsmReader osm = new OsmReader();
osm.references = ref == null ? new DataSet() : ref;
@@ -467,6 +482,6 @@
if (o.id 0)
o.id = 0;
- return osm.ds;
+ return osm;
}
}
___
josm-dev mailing list
josm-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev
