Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi Jon,
I just tried your patch but now I always get the following error when doing
an import:
node_changed_mark failed: ERROR: prepared statement node_changed_mark
does not exist
Any ideas?
Thanks,
Jason
On Fri, Jan 22, 2010 at 8:48 PM, Jason Beverage jasonbever...@gmail.comwrote:
Hi John,
Thanks for the patch, I will give it a test this weekend! I think it would
be a decent option on the command line if it works out well.
Jason
On Fri, Jan 22, 2010 at 6:38 PM, Jon Burgess
jburgess...@googlemail.comwrote:
I'm working with osm2pgsql (latest trunk version) to import OSM into
a
PostgreSQL/PostGIS database. The issue I'm having is that if I use
the --slim option I'll occasionally get an error similar to:
Going over pending relations
COPY_END for COPY osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint
osm_rels_pkey
CONTEXT: COPY osm_rels, line 1207: 2841320323
{42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585...
If I don't use slim mode, it seems to always work. Right now I'm
just
testing with state snapshots from http://downloads.cloudmade.com/,
but
I'd like to be able to import the whole OSM planet database and I'd
really like to be able to use slim mode so I can do incremental
updates and reduce the memory requirements for importing.
If I remember correctly this error occurs when you try to import two
data sets which contain some overlapping data. In your case it appears
that relation ID 284132 appears in both the data extracts. Only the
slim
mode keeps a copy of this node/way/relation data and triggers this
error.
You could try the attached patch to osm2pgsql. This makes it treat all
new data as a modify operation which should avoid problems caused by
duplicate data. Unfortunately this makes the data import process
significantly slower so I won't be applying this change to the trunk
code.
I will think about adding this as a new command line option and try to
figure out if there is something that can be done to reduce the
performance penalty.
Jon
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
I'm working with osm2pgsql (latest trunk version) to import OSM into
a
PostgreSQL/PostGIS database. The issue I'm having is that if I use
the --slim option I'll occasionally get an error similar to:
Going over pending relations
COPY_END for COPY osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint
osm_rels_pkey
CONTEXT: COPY osm_rels, line 1207: 2841320323
{42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585...
If I don't use slim mode, it seems to always work. Right now I'm
just
testing with state snapshots from http://downloads.cloudmade.com/,
but
I'd like to be able to import the whole OSM planet database and I'd
really like to be able to use slim mode so I can do incremental
updates and reduce the memory requirements for importing.
If I remember correctly this error occurs when you try to import two
data sets which contain some overlapping data. In your case it appears
that relation ID 284132 appears in both the data extracts. Only the
slim
mode keeps a copy of this node/way/relation data and triggers this
error.
You could try the attached patch to osm2pgsql. This makes it treat all
new data as a modify operation which should avoid problems caused by
duplicate data. Unfortunately this makes the data import process
significantly slower so I won't be applying this change to the trunk
code.
I will think about adding this as a new command line option and try to
figure out if there is something that can be done to reduce the
performance penalty.
Jon
Index: osm2pgsql.c
===
--- osm2pgsql.c (revision 19148)
+++ osm2pgsql.c (working copy)
@@ -137,7 +137,7 @@
actions_t new_action = ACTION_NONE;
xmlChar *action = xmlTextReaderGetAttribute( reader, BAD_CAST action );
if( action == NULL )
-new_action = ACTION_CREATE;
+new_action = ACTION_MODIFY;
else if( strcmp((char *)action, modify) == 0 )
new_action = ACTION_MODIFY;
else if( strcmp((char *)action, delete) == 0 )
@@ -160,7 +160,7 @@
if (xmlStrEqual(name, BAD_CAST osm))
{
filetype = FILETYPE_OSM;
-action = ACTION_CREATE;
+action = ACTION_MODIFY;
}
else if (xmlStrEqual(name, BAD_CAST osmChange))
{
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi John,
Thanks for the patch, I will give it a test this weekend! I think it would
be a decent option on the command line if it works out well.
Jason
On Fri, Jan 22, 2010 at 6:38 PM, Jon Burgess jburgess...@googlemail.comwrote:
I'm working with osm2pgsql (latest trunk version) to import OSM into
a
PostgreSQL/PostGIS database. The issue I'm having is that if I use
the --slim option I'll occasionally get an error similar to:
Going over pending relations
COPY_END for COPY osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint
osm_rels_pkey
CONTEXT: COPY osm_rels, line 1207: 2841320323
{42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585...
If I don't use slim mode, it seems to always work. Right now I'm
just
testing with state snapshots from http://downloads.cloudmade.com/,
but
I'd like to be able to import the whole OSM planet database and I'd
really like to be able to use slim mode so I can do incremental
updates and reduce the memory requirements for importing.
If I remember correctly this error occurs when you try to import two
data sets which contain some overlapping data. In your case it appears
that relation ID 284132 appears in both the data extracts. Only the
slim
mode keeps a copy of this node/way/relation data and triggers this
error.
You could try the attached patch to osm2pgsql. This makes it treat all
new data as a modify operation which should avoid problems caused by
duplicate data. Unfortunately this makes the data import process
significantly slower so I won't be applying this change to the trunk
code.
I will think about adding this as a new command line option and try to
figure out if there is something that can be done to reduce the
performance penalty.
Jon
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Jan 21, 2010, at 5:51 AM, Mike Warren wrote: Jon Burgess jburgess...@googlemail.com writes: If I remember correctly this error occurs when you try to import two data sets which contain some overlapping data. In your case it appears that relation ID 284132 appears in both the data extracts. Only the slim mode keeps a copy of this node/way/relation data and triggers this error. FWIW, I had a similar error (duplicate node) importing just the alberta.osm.bz2 CloudMade extract from December 25th; non-slim mode works fine. Same error here as well with the haiti extracts from geofabrik today. Dane ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
2010/1/21 Jon Burgess jburgess...@googlemail.com:
On Thu, 2010-01-21 at 11:13 -0500, Jason Beverage wrote:
I'm working with osm2pgsql (latest trunk version) to import OSM into a
PostgreSQL/PostGIS database. The issue I'm having is that if I use
the --slim option I'll occasionally get an error similar to:
Going over pending relations
COPY_END for COPY osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint
osm_rels_pkey
CONTEXT: COPY osm_rels, line 1207: 284132 0 323
{42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585...
If I don't use slim mode, it seems to always work. Right now I'm just
testing with state snapshots from http://downloads.cloudmade.com/, but
I'd like to be able to import the whole OSM planet database and I'd
really like to be able to use slim mode so I can do incremental
updates and reduce the memory requirements for importing.
If I remember correctly this error occurs when you try to import two
data sets which contain some overlapping data. In your case it appears
that relation ID 284132 appears in both the data extracts. Only the slim
mode keeps a copy of this node/way/relation data and triggers this
error.
It is not clear what the code should do in this case. It is possible
that some of the data from the original object might have been removed
when the extract was created because it fell outside the bounding box so
neither of the extracts have the complete information. This issue is
probably easier to understand if you consider what might happen to the
way representing a road crossing the border between two countries.
The only obvious solution is to either manually fix up the overlapping
data or obtain an extract with all the data you want in a single file.
Or, merge the way JOSM merges freshly downloaded data with your
existing layer. This would assume the extracts are created with the
option that keeps the full ways /relations if only one node is inside
the area, like the map API call does.
Cheers,
Andrew
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Thu, 2010-01-21 at 11:29 -0800, Dane Springmeyer wrote: On Jan 21, 2010, at 5:51 AM, Mike Warren wrote: Jon Burgess jburgess...@googlemail.com writes: If I remember correctly this error occurs when you try to import two data sets which contain some overlapping data. In your case it appears that relation ID 284132 appears in both the data extracts. Only the slim mode keeps a copy of this node/way/relation data and triggers this error. FWIW, I had a similar error (duplicate node) importing just the alberta.osm.bz2 CloudMade extract from December 25th; non-slim mode works fine. Same error here as well with the haiti extracts from geofabrik today. It seems there are two versions of a way in the haiti data: $ bzip2 -dc 2010-01-21-19-30.osm.bz2 | grep 28418855 way id=28418855 version=18 timestamp=2010-01-21T05:58:07Z uid=16591 user=neuhausr changeset=3672284 way id=28418855 version=19 timestamp=2010-01-21T11:07:28Z uid=1164 user=dmgroom changeset=3673648 I don't know why they are both there, it maybe a bug in the export process. One idea which was proposed for osm2pgsql was to record and use the version field when applying updates. I'm not sure if this would help in this case, we don't expect to find multiple versions when importing a single file. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Jon Burgess wrote: It seems there are two versions of a way in the haiti data: My bad. I forgot a --simc switch when I moved from Haiti only to Haiti+DomRep this afternoon and this broke things. It is fixed now. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Fri, Jan 22, 2010 at 8:19 AM, Frederik Ramm frede...@remote.org wrote: Hi, Jon Burgess wrote: It seems there are two versions of a way in the haiti data: My bad. I forgot a --simc switch when I moved from Haiti only to Haiti+DomRep this afternoon and this broke things. It is fixed now. If you use the latest SVN version it will detect if multiple versions of an entity exist and throw an error in the --apply-change task. I'll try to do another official release of Osmosis over the next day or two with this change included. This has caught a number of people out now, and it's very non-obvious. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Brett Henderson wrote: If you use the latest SVN version it will detect if multiple versions of an entity exist and throw an error in the --apply-change task. I'll try to do another official release of Osmosis over the next day or two with this change included. Just for laughs - and wonder if you can come up with something better? - this is what I did to remove the duplicate objects from an existing OSM file named faulty.osm: echo osm/osm empty.osm osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm (The two osmosis steps could be written as one but that would probably make it more confusing.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Fri, Jan 22, 2010 at 8:54 AM, Frederik Ramm frede...@remote.org wrote: Hi, Brett Henderson wrote: If you use the latest SVN version it will detect if multiple versions of an entity exist and throw an error in the --apply-change task. I'll try to do another official release of Osmosis over the next day or two with this change included. Just for laughs - and wonder if you can come up with something better? - this is what I did to remove the duplicate objects from an existing OSM file named faulty.osm: echo osm/osm empty.osm osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm (The two osmosis steps could be written as one but that would probably make it more confusing.) I was wondering how you fixed it. That's neat. Do you need the --sort-change-0.6? You probably do, but I don't know what order you end up with using --apply-change with full history diffs. Perhaps I should create a --read-zero task. An empty osm file is often handy. It would avoid having to create empty.osm files like that. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Brett Henderson wrote: I was wondering how you fixed it. That's neat. Do you need the --sort-change-0.6? You probably do, but I don't know what order you end up with using --apply-change with full history diffs. I tried it without, and it complained - for some reason the newer version came before the older version in the file. Perhaps I should create a --read-zero task. An empty osm file is often handy. It would avoid having to create empty.osm files like that. True. Or maybe --read-inline data='osm /' ;-) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Jan 21, 2010, at 1:19 PM, Frederik Ramm wrote: It is fixed now. Bye Frederik Thanks Frederik! Dane ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
