Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Matthias Julius]

Stefan de Konink  writes:

> Frederik Ramm wrote:
>> Stefan de Konink wrote:
>>> So you actually suggest to implement a tree of relations to render a
>>> country?
>> 
>> I'm not convinced a tree is needed.
>
> What do you have in mind to use as area that is bounded by a polygon 
> that has more than 2k of nodes?

A relation with 2000 ways with 2000 nodes each can be an area with
almost 4,000,000 nodes (3,998,000 to be exact).  That will get you
pretty far.

Matthias

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Jason Reid]

Shaun McDonald wrote:
> On 28 Dec 2008, at 20:50, Florian Lohoff wrote:
>
>   
>> On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
>> 
>>> So what is the fundamental problem that your tools break on 40k of
>>> nodes? Bbox them on the request, and return a partial result that  
>>> will
>>> work just as good for rendering.
>>>   
>> The db schema uses smallint for the node number on the way - so you  
>> may
>> only address -32768 to +32767 number of nodes on a way - 40k nodes
>> is out of scope of a smallint so osmosis breaks on importing the
>> changeset.
>> 
>
>  From API 0.6 there is a limit of 2,000 nodes in a way.
> http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225
> and
> http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9
>
> Hopefully someone will fix the way prior to the new api transition.
>
> Shaun
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>   
There are actually a number of ways that are excessively too large 
currently, many of the largest are geopolitical boundaries from the 
Geobase import (the import was 162,000 nodes in around 500 ways). We've 
been discussing how to reduce them into more manageable chunks over on 
talk-ca. So they will all be under the limit by the time the transition 
happens.

-Jason Reid

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Jeremy Adams wrote:
> I ran the same command on my box about 4 hours ago and it's still
> cranking away.  Should it really take that long to change a column
> type?  It's got one of the CPU cores pegged, so I assume it's still
> doing something.

It should not do anything on your CPU; such actions are disk operations. 
Depending on the DBS it is possible that it is faster to copy a column 
and merge it back later than to alter.

Does your column have constraints on it?


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Jeremy Adams]

I ran the same command on my box about 4 hours ago and it's still cranking 
away.  Should it really take that long to change a column type?  It's got one 
of the CPU cores pegged, so I assume it's still doing something.

-Jeremy


Original Message ---
On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote:
> osm=> select max(sequence_id) from way_nodes;
>   max
>   ---
>39767
>   (1 row)
> 
> osm=> select * from way_nodes where sequence_id = 39767;
>   way_id  |  node_id  | sequence_id
>   --+---+-
>28098452 | 308532457 |   39767
> 
> I converted the smallint to int ...
> 
> Flo
> 
> So, perhaps following Flo's lead and changing the smallint to int would be
> the best approach for dealing with getting the ROMA servers back up and
> running...

Both of my databases broke down again as i dropped the databases and
recreated them - right now converting again ...

I have no clue if this solves it - forgot already what i did last time ;)

ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer;

Hopefully postgres is clever enough to take over the old content and
recreate the index ...

Flo
-- 
Florian Lohoff  f...@rfc822.org +49-171-2280134
Those who would give up a little freedom to get a little 
  security shall soon have neither - Benjamin Franklin


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Brett Henderson]

Seems like you're all on top of the problem now.

I updated the schema creation script on 3rd November to use an int 
instead of smallint.  Internally osmosis doesn't care, it uses a 32 bit 
integer (more accurately, it uses a list index which is 32 bit).  It is 
just the schema itself which was using a smallint to minimise disk 
usage.  It should be possible to change the column type without losing 
data.  If there's anything I can do let me know.

Note that the 0.6 version of the schema still uses a smallint, I 
possibly should change that in case people start testing it with data 
migrated from a 0.5 planet.

D Tucny wrote:
> 2008/12/29 Jeremy Adams  >
>
> Hey all,
>
> There's an issue with the minute change file
> 200812290912-200812290913.osc.gz.
> It contains a way which has over 40k nodes.  It's an
> administrative boundary
> which appears to enclose all of Quebec.  It's way id is 29309772.
>  When osmosis
> attempts to process this file it fails.  The relevant parts of the
> error appear
> to be "Unable to insert new way node" and a pgsql error "ERROR:
> smallint out of
> range."
>
> I'm sure I can find the way and split it into smaller chunks, but
> what's the
> process for getting the ROMA servers running again?  Can we just
> remove the
> reference to the way in the changeset and continue updating?  The
> way should be
> added back in once it's split and reuploaded, correct?
>
> -Jeremy
>
>
> The same problem happened on 31st October...
>
> Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple 
> shema / smallint out of range
>
> On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote:
>
> > Hi,
> > i just discovered that osmosis was not able to apply the hourly osc file
> > starting 2008-10-29T20:00:00Z - It failed with:
> >
> > 2008-10-31 11:09:52 CET ERROR:  smallint out of range
> > 2008-10-31 11:09:52 CET STATEMENT:  INSERT INTO way_nodes (way_id, 
> node_id, sequence_id) VALUES ($1, $2, $3)
> >
> > Is there a way with >2^16 aka 65536 nodes?
> >
> > Or did someone manage to enter a completely broken sequence number?
> ... [show rest of quote 
> ]
> ... [rest of quote removed]
> Its 2^15 because it signed - and yes - somebody managed to get abovE:
>
> osm=> select max(sequence_id) from way_nodes;
>   max  
>   ---
>39767
>   (1 row)
>
> osm=> select * from way_nodes where sequence_id = 39767;
>   way_id  |  node_id  | sequence_id
>   --+---+-
>28098452 | 308532457 |   39767
>
> I converted the smallint to int ...
>
> Flo 
>
> So, perhaps following Flo's lead and changing the smallint to int 
> would be the best approach for dealing with getting the ROMA servers 
> back up and running...
>
> d
>
> 
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>   


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[milenko]

What is the TRAPI server? (besides apparently very quick?)  Any specs/munin 
graphs for the curious?

It's pretty much handling the load all by itself today.

-Jeremy

Original Message ---
+--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams 
wrote:
| It's not the ROMA script that's failing - osmosis is failing while it's
| trying to update the database using changesets from the main API.
|
| Whatever db schema the main API uses must allow for this, while the one
| for osmosis that was used to setup at least my server does not.

I don't really see why it would be a db schema problem (it didn't fail on
my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3
fields :-)

Also, I've added the TRAPI server to the ROMA load balancer because well,
there was only me for a few hours, and it was making me nervous, which was
a good thing, because my ROMA is down now. (A drive earlier today failed,
I'll try to fix it in a week when I come back from holiday.)

--
Mathieu Arnold


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Frederik Ramm wrote:
> Stefan de Konink wrote:
>> So you actually suggest to implement a tree of relations to render a
>> country?
> 
> I'm not convinced a tree is needed.

What do you have in mind to use as area that is bounded by a polygon 
that has more than 2k of nodes?

>> Totally break lookups such as 'is X in country Y', you will
>> make friends that is for sure :)
> 
> Someone who is so quick with taking complex programming for granted 
> ("... simply implement partial editing of ways...") will surely not be 
> foiled by the requirement to access a second level of relations which 
> seems rather trivial in comparison.

I do what everyone does that is smart, not reinvent the wheel. Geos is 
for us geometry-dropouts. There is no a priori need to break things up. 
If you do want to break things up, implement server/clientside what I 
suggested before, partial ways (as in return less nodes) limited by 
bboxing. That has a effect on any form of processing.

>> So you really want people first to change everything to 0.6 and then 
>> to something else ;) I wanted to write longer about this, but I think 
>> allowing everyone to send just a complete changeset (within one 
>> request) ditch REST for updating will give much more momentum, and 
>> requires less code.
> 
> Thank you but we need programmers, not advice. Sending a complete 
> changeset within one request is possible BTW.

You have already implemented it; (I have too btw). Since I'm not going 
to code Ruby or Java 'for fun', I'll stick with the C implementation of 
the API. Like other people work on other editors not in Java.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Shaun McDonald wrote:
>> So you actually suggest to implement a tree of relations to render a
>> country? Totally break lookups such as 'is X in country Y', you will
>> make friends that is for sure :)
> 
> That is not what relations are for. You determine which country is in 
> based on the latitude and longitude of the nodes of the items, and a 
> bounding polygon for the country that you are interested in.

That polygon just happens to be the cutted way/relation, or whatever 
category you want to assign to it. Sence it is not one thing anymore, it 
is not cyclic, thus broken thus mathematically extremely inefficient to 
calculate upon.

>>> It won't. Partial modification of ways is a good idea but we lack the
>>> coders to implement it in API and clients. It might be done at a
>>> later time but surely not with 0.6.
>>
>> So you really want people first to change everything to 0.6 and then 
>> to something else ;) I wanted to write longer about this, but I think 
>> allowing everyone to send just a complete changeset (within one 
>> request)  ditch REST for updating will give much more momentum, and 
>> requires less code.
> 
> One step at a time.

(As mentioned; it is already in there...)


Stefan


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Frederik Ramm]

Hi,

Stefan de Konink wrote:
> So you actually suggest to implement a tree of relations to render a
> country?

I'm not convinced a tree is needed.

> Totally break lookups such as 'is X in country Y', you will
> make friends that is for sure :)

Someone who is so quick with taking complex programming for granted 
("... simply implement partial editing of ways...") will surely not be 
foiled by the requirement to access a second level of relations which 
seems rather trivial in comparison.

> So you really want people first to change everything to 0.6 and then to 
> something else ;) I wanted to write longer about this, but I think 
> allowing everyone to send just a complete changeset (within one request) 
> ditch REST for updating will give much more momentum, and requires less 
> code.

Thank you but we need programmers, not advice. Sending a complete 
changeset within one request is possible BTW.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Shaun McDonald]

On 28 Dec 2008, at 22:10, Stefan de Konink wrote:

> Frederik Ramm wrote:
>> Stefan de Konink wrote:
>>> Shaun McDonald wrote:
 From API 0.6 there is a limit of 2,000 nodes in a way.
>>> Btw; Do you impose this also on a relation?
>> Yes.
>
> So you actually suggest to implement a tree of relations to render a
> country? Totally break lookups such as 'is X in country Y', you will
> make friends that is for sure :)

That is not what relations are for. You determine which country is in  
based on the latitude and longitude of the nodes of the items, and a  
bounding polygon for the country that you are interested in.

>
>
>> It won't. Partial modification of ways is a good idea but we lack the
>> coders to implement it in API and clients. It might be done at a
>> later time but surely not with 0.6.
>
> So you really want people first to change everything to 0.6 and then  
> to something else ;) I wanted to write longer about this, but I  
> think allowing everyone to send just a complete changeset (within  
> one request)  ditch REST for updating will give much more momentum,  
> and requires less code.

One step at a time.

Shaun


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Stefan de Konink wrote:
> So you really want people first to change everything to 0.6 and then to 
> something else ;) I wanted to write longer about this, but I think 
> allowing everyone to send just a complete changeset (within one request) 
>   ditch REST for updating will give much more momentum, and requires 
> less code.

POST changeset/#id/uploadUploads a diff into a changeset 
transactionally. Returns XML with the committed versions and IDs of the 
diff elements.   Coded, Tested

Looking at the specs that is already implemented. ..


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Frederik Ramm wrote:
> Stefan de Konink wrote:
>> Shaun McDonald wrote:
>>> From API 0.6 there is a limit of 2,000 nodes in a way.
>> 
>> Btw; Do you impose this also on a relation?
> 
> Yes.

So you actually suggest to implement a tree of relations to render a
country? Totally break lookups such as 'is X in country Y', you will
make friends that is for sure :)

> It won't. Partial modification of ways is a good idea but we lack the
> coders to implement it in API and clients. It might be done at a
> later time but surely not with 0.6.

So you really want people first to change everything to 0.6 and then to 
something else ;) I wanted to write longer about this, but I think 
allowing everyone to send just a complete changeset (within one request) 
  ditch REST for updating will give much more momentum, and requires 
less code.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Frederik Ramm]

Hi,

Stefan de Konink wrote:
> Shaun McDonald wrote:
>>  From API 0.6 there is a limit of 2,000 nodes in a way.
> 
> Btw; Do you impose this also on a relation? 

Yes.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Frederik Ramm]

Hi,

Stefan de Konink wrote:
> Shaun McDonald wrote:
>>  From API 0.6 there is a limit of 2,000 nodes in a way.
> 
> ...and I hope this limit will be dropped again dropped 

It won't. Partial modification of ways is a good idea but we lack the 
coders to implement it in API and clients. It might be done at a later 
time but surely not with 0.6.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Florian Lohoff wrote:
> Would be kind of complicated i guess.
> 
> The point is that you have number 1..100 on a  way - you edit the range
> from 50-60 and insert 5 nodes than you'll have 50-65 which will have an
> overlap with the ones in the database (sequence wise) so the database/api
> needs to reorganize sequence numbers of nodes you didnt edit. Id say -
> very ugly ..

You would get the current version of that way back when you download. A 
subtree can be seen as ordered; for sake of redundancy we will add the 
sequence of moment of serving.

Using the locking assumption we could just add remove the ways that 
where added or removed from the relation by the fantastic model that 
Microsoft Research has thought out (it is not a joke) ..

 to insert a node after the first node.
 to insert a node before the first node.

Now this would work out pretty well, aka server verifies that the 
current changeset equals the submitted changes and inserts the nodes, 
and reorders the index.


But now what if we do have transactions;
- Server based best effort, like diff does with fuzzy matching
- Just tell the user you are too late this is the way now, and let your 
editor of choice fix the mess.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Florian Lohoff]

On Sun, Dec 28, 2008 at 09:15:34PM +0100, Stefan de Konink wrote:
> Subject: Re: [OSM-dev] ROMA servers down - osmosis large way problem
> 
> Shaun McDonald wrote:
> > From API 0.6 there is a limit of 2,000 nodes in a way.
> 
> ...and I hope this limit will be dropped again dropped and allow partial 
> modification for ways opposed to uploading the entire way.

Would be kind of complicated i guess.

The point is that you have number 1..100 on a  way - you edit the range
from 50-60 and insert 5 nodes than you'll have 50-65 which will have an
overlap with the ones in the database (sequence wise) so the database/api
needs to reorganize sequence numbers of nodes you didnt edit. Id say -
very ugly ..

Flo
-- 
Florian Lohoff  f...@rfc822.org +49-171-2280134
Those who would give up a little freedom to get a little 
  security shall soon have neither - Benjamin Franklin


signature.asc
Description: Digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Shaun McDonald wrote:
>  From API 0.6 there is a limit of 2,000 nodes in a way.

Btw; Do you impose this also on a relation? If not, then it is pretty 
useless to have an infrastructure that is capable of holding an infinite 
amount of nodes and one that is not ;)


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Shaun McDonald wrote:
>  From API 0.6 there is a limit of 2,000 nodes in a way.

...and I hope this limit will be dropped again dropped and allow partial 
modification for ways opposed to uploading the entire way.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Shaun McDonald]

On 28 Dec 2008, at 20:50, Florian Lohoff wrote:

> On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
>> So what is the fundamental problem that your tools break on 40k of
>> nodes? Bbox them on the request, and return a partial result that  
>> will
>> work just as good for rendering.
>
> The db schema uses smallint for the node number on the way - so you  
> may
> only address -32768 to +32767 number of nodes on a way - 40k nodes
> is out of scope of a smallint so osmosis breaks on importing the
> changeset.

 From API 0.6 there is a limit of 2,000 nodes in a way.
http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/app/models/way.rb#L225
and
http://trac.openstreetmap.org/browser/sites/rails_port_branches/api06/config/application.yml#L9

Hopefully someone will fix the way prior to the new api transition.

Shaun


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Shaun McDonald]

On 28 Dec 2008, at 20:56, Mathieu Arnold wrote:

> +--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff 
> wrote:
> | On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
> |> So what is the fundamental problem that your tools break on 40k of
> |> nodes? Bbox them on the request, and return a partial result that  
> will
> |> work just as good for rendering.
> |
> | The db schema uses smallint for the node number on the way - so  
> you may
> | only address -32768 to +32767 number of nodes on a way - 40k nodes
> | is out of scope of a smallint so osmosis breaks on importing the
> | changeset.
>
> I'm not certain what we're talking about, but in the version of  
> osmosis I
> found, the pgsql_simple_schema_0.5.sql file contains :
> CREATE TABLE way_nodes (
>way_id bigint NOT NULL,
>node_id bigint NOT NULL,
>sequence_id int NOT NULL
> );
>

I think this is a problem that was fixed the last time, though the  
people running the ROMA servers haven't updated to the latest svn  
version of osmosis to get the latest updates.

Shaun


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[milenko]

Weird - I checked the schema that I used and it's a smallint.  That must have 
been changed recently.

I've altered the column type on my server - we'll see what happens when it's 
finished.

-Jeremy

Original Message ---
+--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff 
wrote:
| On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
|> So what is the fundamental problem that your tools break on 40k of
|> nodes? Bbox them on the request, and return a partial result that will
|> work just as good for rendering.
|
| The db schema uses smallint for the node number on the way - so you may
| only address -32768 to +32767 number of nodes on a way - 40k nodes
| is out of scope of a smallint so osmosis breaks on importing the
| changeset.

I'm not certain what we're talking about, but in the version of osmosis I
found, the pgsql_simple_schema_0.5.sql file contains :
CREATE TABLE way_nodes (
way_id bigint NOT NULL,
node_id bigint NOT NULL,
sequence_id int NOT NULL
);


--
Mathieu Arnold

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Florian Lohoff wrote:
> On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
>> So what is the fundamental problem that your tools break on 40k of 
>> nodes? Bbox them on the request, and return a partial result that will 
>> work just as good for rendering.
> 
> The db schema uses smallint for the node number on the way - so you may
> only address -32768 to +32767 number of nodes on a way - 40k nodes 
> is out of scope of a smallint so osmosis breaks on importing the
> changeset.

Ok fundamental problem: premature optimisations. How much disk space 
have been saved with this optimisation until now? (With an alter table 
you would be able to see it.)


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Mathieu Arnold]

+--On 28 décembre 2008 20:50:00 +0100 Florian Lohoff 
wrote:
| On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
|> So what is the fundamental problem that your tools break on 40k of 
|> nodes? Bbox them on the request, and return a partial result that will 
|> work just as good for rendering.
| 
| The db schema uses smallint for the node number on the way - so you may
| only address -32768 to +32767 number of nodes on a way - 40k nodes 
| is out of scope of a smallint so osmosis breaks on importing the
| changeset.

I'm not certain what we're talking about, but in the version of osmosis I
found, the pgsql_simple_schema_0.5.sql file contains :
CREATE TABLE way_nodes (
way_id bigint NOT NULL,
node_id bigint NOT NULL,
sequence_id int NOT NULL
);


-- 
Mathieu Arnold

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Florian Lohoff]

On Mon, Dec 29, 2008 at 03:43:27AM +0800, D Tucny wrote:
> osm=> select max(sequence_id) from way_nodes;
>   max
>   ---
>39767
>   (1 row)
> 
> osm=> select * from way_nodes where sequence_id = 39767;
>   way_id  |  node_id  | sequence_id
>   --+---+-
>28098452 | 308532457 |   39767
> 
> I converted the smallint to int ...
> 
> Flo
> 
> So, perhaps following Flo's lead and changing the smallint to int would be
> the best approach for dealing with getting the ROMA servers back up and
> running...

Both of my databases broke down again as i dropped the databases and
recreated them - right now converting again ...

I have no clue if this solves it - forgot already what i did last time ;)

ALTER TABLE way_nodes ALTER COLUMN sequence_id TYPE integer;

Hopefully postgres is clever enough to take over the old content and
recreate the index ...

Flo
-- 
Florian Lohoff  f...@rfc822.org +49-171-2280134
Those who would give up a little freedom to get a little 
  security shall soon have neither - Benjamin Franklin


signature.asc
Description: Digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Florian Lohoff]

On Sun, Dec 28, 2008 at 08:25:54PM +0100, Stefan de Konink wrote:
> So what is the fundamental problem that your tools break on 40k of 
> nodes? Bbox them on the request, and return a partial result that will 
> work just as good for rendering.

The db schema uses smallint for the node number on the way - so you may
only address -32768 to +32767 number of nodes on a way - 40k nodes 
is out of scope of a smallint so osmosis breaks on importing the
changeset.

Flo
-- 
Florian Lohoff  f...@rfc822.org +49-171-2280134
Those who would give up a little freedom to get a little 
  security shall soon have neither - Benjamin Franklin


signature.asc
Description: Digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Mathieu Arnold]

+--On 28 décembre 2008 14:31:40 -0500 Jeremy Adams 
wrote:
| It's not the ROMA script that's failing - osmosis is failing while it's
| trying to update the database using changesets from the main API.
| 
| Whatever db schema the main API uses must allow for this, while the one
| for osmosis that was used to setup at least my server does not.

I don't really see why it would be a db schema problem (it didn't fail on
my ROMA), it's not as if the ways table has nd1, nd2, nd3, ... nd3
fields :-)

Also, I've added the TRAPI server to the ROMA load balancer because well,
there was only me for a few hours, and it was making me nervous, which was
a good thing, because my ROMA is down now. (A drive earlier today failed,
I'll try to fix it in a week when I come back from holiday.)

-- 
Mathieu Arnold

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[D Tucny]

2008/12/29 Jeremy Adams 

> Hey all,
>
> There's an issue with the minute change file
> 200812290912-200812290913.osc.gz.
> It contains a way which has over 40k nodes.  It's an administrative
> boundary
> which appears to enclose all of Quebec.  It's way id is 29309772.  When
> osmosis
> attempts to process this file it fails.  The relevant parts of the error
> appear
> to be "Unable to insert new way node" and a pgsql error "ERROR: smallint
> out of
> range."
>
> I'm sure I can find the way and split it into smaller chunks, but what's
> the
> process for getting the ROMA servers running again?  Can we just remove the
> reference to the way in the changeset and continue updating?  The way
> should be
> added back in once it's split and reuploaded, correct?
>
> -Jeremy
>

The same problem happened on 31st October...

Subject: HEADS UP osmosis pgsql schema users Was: psql osmosis simple shema
/ smallint out of range

On Fri, Oct 31, 2008 at 11:17:16AM +0100, Florian Lohoff wrote:
> Hi,
> i just discovered that osmosis was not able to apply the hourly osc file
> starting 2008-10-29T20:00:00Z - It failed with:
>
> 2008-10-31 11:09:52 CET ERROR:  smallint out of range
> 2008-10-31 11:09:52 CET STATEMENT:  INSERT INTO way_nodes (way_id,
node_id, sequence_id) VALUES ($1, $2, $3)
>
> Is there a way with >2^16 aka 65536 nodes?
>
> Or did someone manage to enter a completely broken sequence number?
... [show rest of
quote
]
... [rest of quote removed]
Its 2^15 because it signed - and yes - somebody managed to get abovE:

osm=> select max(sequence_id) from way_nodes;
  max
  ---
   39767
  (1 row)

osm=> select * from way_nodes where sequence_id = 39767;
  way_id  |  node_id  | sequence_id
  --+---+-
   28098452 | 308532457 |   39767

I converted the smallint to int ...

Flo

So, perhaps following Flo's lead and changing the smallint to int would be
the best approach for dealing with getting the ROMA servers back up and
running...

d
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Jeremy Adams wrote:
> It's not the ROMA script that's failing - osmosis is failing while
> it's trying to update the database using changesets from the main
> API.
> 
> Whatever db schema the main API uses must allow for this, while the
> one for osmosis that was used to setup at least my server does not.

Ok; I cannot understand the difficulty in it :{ I mean it is just the 
parsing of a subtree.

My own update mechanism is rather untested. Otherwise I would suggest 
that. It basically allows any user to submit changesets as part of their 
edits. If my relative resolver is done I'll propose it as alternative 
API mechanism that doesn't care about changesets anymore because it is a 
changeset by itself.

(people that are interested in it know where to look)


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Jeremy Adams]

It's not the ROMA script that's failing - osmosis is failing while it's trying 
to update the database using changesets from the main API.

Whatever db schema the main API uses must allow for this, while the one for 
osmosis that was used to setup at least my server does not.

-Jeremy

Original Message ---
Jeremy Adams wrote:
> There's an issue with the minute change file 200812290912-200812290913.osc.gz.
> It contains a way which has over 40k nodes.  It's an administrative boundary
> which appears to enclose all of Quebec.  It's way id is 29309772.  When 
> osmosis
> attempts to process this file it fails.  The relevant parts of the error 
> appear
> to be "Unable to insert new way node" and a pgsql error "ERROR: smallint out 
> of
> range."
> 
> I'm sure I can find the way and split it into smaller chunks, but what's the
> process for getting the ROMA servers running again?  Can we just remove the
> reference to the way in the changeset and continue updating?  The way should 
> be
> added back in once it's split and reuploaded, correct?

So what is the fundamental problem that your tools break on 40k of 
nodes? Bbox them on the request, and return a partial result that will 
work just as good for rendering.


Stefan


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ROMA servers down - osmosis large way problem

[Stefan de Konink]

Jeremy Adams wrote:
> There's an issue with the minute change file 200812290912-200812290913.osc.gz.
> It contains a way which has over 40k nodes.  It's an administrative boundary
> which appears to enclose all of Quebec.  It's way id is 29309772.  When 
> osmosis
> attempts to process this file it fails.  The relevant parts of the error 
> appear
> to be "Unable to insert new way node" and a pgsql error "ERROR: smallint out 
> of
> range."
> 
> I'm sure I can find the way and split it into smaller chunks, but what's the
> process for getting the ROMA servers running again?  Can we just remove the
> reference to the way in the changeset and continue updating?  The way should 
> be
> added back in once it's split and reuploaded, correct?

So what is the fundamental problem that your tools break on 40k of 
nodes? Bbox them on the request, and return a partial result that will 
work just as good for rendering.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] ROMA servers down - osmosis large way problem

[Jeremy Adams]

Hey all,

There's an issue with the minute change file 200812290912-200812290913.osc.gz.
It contains a way which has over 40k nodes.  It's an administrative boundary
which appears to enclose all of Quebec.  It's way id is 29309772.  When osmosis
attempts to process this file it fails.  The relevant parts of the error appear
to be "Unable to insert new way node" and a pgsql error "ERROR: smallint out of
range."

I'm sure I can find the way and split it into smaller chunks, but what's the
process for getting the ROMA servers running again?  Can we just remove the
reference to the way in the changeset and continue updating?  The way should be
added back in once it's split and reuploaded, correct?

-Jeremy


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What about moving wmsplugin/webkit-image to a pplication/utils? Was: Need help compiling wmsplugin /webkit-image.cpp on Debian-lenny

[Joerg Ostertag (OSM Tettnang/Germany)]

On Sonntag 28 Dezember 2008, Joerg Ostertag (OSM Tettnang/Germany) wrote:
> On Donnerstag 25 Dezember 2008, Simon Ward wrote:
> > On Thu, Dec 25, 2008 at 08:32:04AM +0100, Joerg Ostertag (OSM
>
> Tettnang/Germany) wrote:
> > > Hi,
> > >
> > > I need help compiling wmsplugin/webkit-image.cpp on my Debian-lenny
> > > machine. when I try to compile I get the following errors:

...

> It works :-)
>
> I changed the Makefile and now I compiled the package with the new
> Makefile. For now I compiled it into the openstreetmap-josm package.


What about moving the webkit-image stuff into the utils tree?
This way I would compile the architecture dependent things in the 
debian-package openstreetmap-utils and there would not be any need to split 
the openstreetmap-josm package into different packages for different 
cpu-types.

-- 
Jörg (Germany, Tettnang)

http://www.ostertag.name/

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Your whishlist for Debian Packages?

[Joerg Ostertag (OSM Tettnang/Germany)]

Hi,

as most of you probably know I'm providing debian packages for the most common 
tools needed to work with OpenStreetMap. 
What I'd like to know is:
   - which additional tools from the osm-svn would you like to see added to
 the debian repository?
   - which other debian based platforms would you like to see suported?
 examples would be: debian-etch, Ubuntu-xx, ...

If you give me a preferences list, I can start with the most wanted ones 

More description on the existing debian repository can be found at:
http://www.gpsdrive.de/development/debian.shtml

-- 
Jörg (Germany, Tettnang)

http://www.ostertag.name/

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Success: Need help compiling wmsplugin/webk it-image.cpp on Debian-lenny

[Joerg Ostertag (OSM Tettnang/Germany)]

On Donnerstag 25 Dezember 2008, Simon Ward wrote:
> On Thu, Dec 25, 2008 at 08:32:04AM +0100, Joerg Ostertag (OSM 
Tettnang/Germany) wrote:
> > Hi,
> >
> > I need help compiling wmsplugin/webkit-image.cpp on my Debian-lenny
> > machine. when I try to compile I get the following errors:
>
> The Makefile in svn for some reason didn’t work for me, and neither did
> the command line on the wiki page.  I ended up with this command line:
>
>   g++ -W -o webkit-image -I/usr/include/qt4/ -O2 $(pkg-config --libs
> --cflags QtCore QtGui QtWebKit) webkit-image.cpp

It works :-)

I changed the Makefile and now I compiled the package with the new Makefile.
For now I compiled it into the openstreetmap-josm package.

-- 
Jörg (Germany, Tettnang)

http://www.ostertag.name/

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] which Java Verison to use (1.6 vs. 1.5); was: Error compiling osmosis

[Andreas Kalsch]

It is OK that Osmosis supports just 1.6, because it has no UI (has it?). 
So there are 2 reasons:

- You use it more on servers with Linux, so 1.6 is supported.
- The user has the competence to deal with this kind of things, 
including setting up his own DB.

JOSM - in opposite - should be as easy as possible to use (= downloading 
and starting to edit), so that as many people as possible contribute to OSM.

So focus on new features and don't try to support ancient versions.

Best,

Andi


Brett Henderson schrieb:
> Sending this to dev because I'm curious to hear thoughts from the JOSM guys.
>
> I agree it would be nice to standardise but the reasons for the 
> difference as I understand it are:
> JOSM is an end user tool where wide platform support is necessary.  Java 
> 1.5 is more widely available than Java 1.6.  OSX in particular hasn't 
> had support for 1.6, not sure if that's changed yet.  JOSM is 
> maintaining compatibility with Java 1.5.  *However* it should be 
> possible to compile JOSM on Java 1.6, it just won't run on 1.5 if you do 
> so.  There may be some warnings around the use of the @Override 
> annotation (I don't know the details here) but I don't think these 
> should be show stoppers.
> Osmosis is newer than JOSM, and is less end-user focused.  I don't 
> support 1.5 because I'm using some newer features of the 1.6 platform.  
>  From memory, these are Java 2D libraries required for accurate polygon 
> support, concurrent libraries, and collection libraries.  Of these it 
> might be possible to support 1.5 with some additional effort but the 
> polygon support in particular is hard to do properly on 1.5 because 1.6 
> added "double" accuracy 2D calculations.
>
> Here's my suggestion.  OpenJDK is a Java 1.6 platform.  I assume that is 
> what Debian will provide out of the box.  Compile both JOSM and Osmosis 
> using this 1.6 platform.  They will both then run on the OpenJDK 
> provided by Debian.  Neither will run on older 1.5 JDKs.  You aren't 
> building binary distributions of JOSM for cross-platform use therefore 
> you don't need to support 1.5.
>
> 1.6 provides some very useful features, it has been released for around 
> 2 years now, and is supported on the vast majority of platforms out 
> there.  Supporting 1.5 just isn't a high priority for me.  With the open 
> source OpenJDK out there I was hoping the need to support ancient java 
> releases would be eliminated.
>
> Thoughts welcome.
> Brett
>
> Joerg Ostertag (OSM Tettnang/Germany) wrote:
>   
>> OK ...
>>
>>   so josm doesn't want java 1.6
>> and osmosis doesn't want java 1.5
>>
>> This doesn't really make it easy to create debian packages out of these ...
>>
>> Couldn't we agree on one Version of Java for all OSM Software. This would 
>> macke packaging much easier for me.
>>
>> -
>>
>> Joerg
>>
>>
>> On Mittwoch 24 Dezember 2008, you wrote:
>>   
>> 
>>> The minimum required java version for osmosis is 1.6.  You appear to be
>>> using version 1.5.
>>> 
>>>   
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>
>   


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] which Java Verison to use (1.6 vs. 1.5); was: Error compiling osmosis

[Shaun McDonald]

On 28 Dec 2008, at 00:36, Brett Henderson wrote:

>  Java
> 1.5 is more widely available than Java 1.6.  OSX in particular hasn't
> had support for 1.6, not sure if that's changed yet.

The current status of Java 1.6 on Mac OS X is that Apple have decided  
to only support it on the Intel 64 bit platform. PPC and Intel 32 bit  
have been left in the cold by Apple.

There is also a bug that Apple have acknowledged, that means that ant  
will not find Java 1.6, using the default mechanisms supplied by  
Apple, where Java 1.6 can be installed. Hopefully the next version of  
the developer tools will have this fixed.

Shaun


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev