Re: [OSM-dev] problem with changesets and bulk_upload.py
2009/7/6 Mikel Maron mikel_ma...@yahoo.com: Recently uploaded the Arabic names for all countries. Rather than updating the existing nodes as expected, it created new nodes. http://www.openstreetmap.org/browse/changeset/1707270 http://www.openstreetmap.org/browse/changeset/1725949 I used bulk_upload.py, processing an osmChange file, with all the nodes wrapped inside a modify tag. Does this not work as expected??! How should it be done? Don't use bulk_upload.py would be a good start. From looking at the code it doesn't take osm change files at all -- it takes JOSM style osm files, which it assumes contains entirely original content. It then proceeds to do no input validation and arbitrarily tree walks the XML file you give it looking for node way relation elements to upload by creating new objects if they have no action attribute, or it doesn't say modify or delete, and doing modifies and deletes where the action attribute says to. Basically it looks like it was hacked up to do a particular job, by someone who knew how it worked and therefore didn't bother with all of the fun you need when you let something like that loose with the population at large. Once the proper method is sorted out, I'll delete these duplicate nodes. So you have 5 choices: - rewrite your input into the format expected by bulk_upload.py (and test it on a dev server or something first) - rewrite bulk_upload.py to not be so crap - use bulk_upload.php (also doesn't take osmChange files though) - write your own bulk uploader that takes osmChange - wait till someone else does that (I was working on one but got distracted by potlatch2) Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] problem with changesets and bulk_upload.py
2009/7/7 Dave Stubbs osm.l...@randomjunk.co.uk: 2009/7/6 Mikel Maron mikel_ma...@yahoo.com: Once the proper method is sorted out, I'll delete these duplicate nodes. So you have 5 choices: - rewrite your input into the format expected by bulk_upload.py (and test it on a dev server or something first) - rewrite bulk_upload.py to not be so crap - use bulk_upload.php (also doesn't take osmChange files though) - write your own bulk uploader that takes osmChange - wait till someone else does that (I was working on one but got distracted by potlatch2) http://www.openstreetmap.pl/balrog/bulkupload/ does this. Cheers ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Mapnik doesn't find geometry columns
I'm running mapnik version 0.5.1 as shipped with Debian Lenny. Following
http://wiki.openstreetmap.org/wiki/Mapnik/Installation#For_Debian_Lenny_or_Sid,
this should be ready to work.
Also, I have downloaded the latest SVN revision from
http://svn.openstreetmap.org/applications/rendering/mapnik and done the
steps described on
http://wiki.openstreetmap.org/wiki/Mapnik#Rendering_with_Mapnik.
I have imported the extract for the UK
(http://nick.dev.openstreetmap.org/downloads/planet/uk-090701.osm.bz2)
with osm2pgsql. No problem with the import and everything looks good in
the PostGIS db. The following tables are populated with reasonably
looking data:
Schema |Name| Type | Owner
--++--+
public | planet_osm_line| table | hansm
public | planet_osm_nodes | table | hansm
public | planet_osm_point | table | hansm
public | planet_osm_polygon | table | hansm
public | planet_osm_rels| table | hansm
public | planet_osm_roads | table | hansm
public | planet_osm_ways| table | hansm
osm= SELECT * FROM geometry_columns;
f_table_catalog | f_table_schema |f_table_name|
f_geometry_column | coord_dimension | srid |type
-+++---+-++
| public | planet_osm_point |
way | 2 | 900913 | POINT
| public | planet_osm_line|
way | 2 | 900913 | LINESTRING
| public | planet_osm_polygon |
way | 2 | 900913 | POLYGON
| public | planet_osm_roads |
way | 2 | 900913 | LINESTRING
When trying to render tiles with generate_tiles, I only get PNG files
with waterbodies and land, but not the actual OSM features. This is
independend from the zoom level.
When checking my PostgreSQL log files, I see malformed SQL queries
issued by mapnik. Obviously, it does not know about the tables' geometry
columns:
select asbinary() as geom,highway,railway from
(select way,highway,railway,
case when tunnel in ('yes','true','1') then 'yes'::text
else tunnel end as tunnel
from planet_osm_roads
where highway is not null
or railway is not null
order by z_order
) as roads
where setSRID('BOX3D(-19724422.27493316
7200979.560689881,-18472078.00350883 8453323.832114212)'::box3d,537534573)
Now my big question: How do I explain mapnik that it has to use way as
geometry column?
Thanks for helping
Hans
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapnik doesn't find geometry columns
Sure, exactly this was the problem! That's why I wanted to know how to
tell Mapnik what the geometry column is. It didn't find it by itself and
just left it blank in the SQL query.
Hans
Milo van der Linden wrote:
Shouldn't the syntax be: select asbinary(way) because I think asbinary()
will return nothing.
?
Hans Musil wrote:
No, this not the problem.
osm= SELECT * FROM spatial_ref_sys WHERE srid = 900913;
shows the corresponding row. Anyhow, I could not imagine that a
missing spatial_ref_sys entry would cause mapnik to forget the name of
the geometry column.
I also have tried a
Parameter name=geometry_fieldway/Parameter
inside each Layer tag in osm.xml, but this seems not yet to work with
version 0.5.1.
Is there a way to run mapnik in debug mode in order to see what
queries it sends to PostgreSQL? I wonder if it really does the SELECT
on the geometry_columns tabel.
Hans
Milo van der Linden wrote:
The table Geometry_columns tells mapnik that it has to use way as the
geometry_column. The only thing that might be wrong is that you do not
have the 900913 projection set up in the spatial_ref_sys table. This
means that features cannot get rendered by mapnik because it does not
know how to project them.
Hans Musil wrote:
I'm running mapnik version 0.5.1 as shipped with Debian Lenny.
Following
http://wiki.openstreetmap.org/wiki/Mapnik/Installation#For_Debian_Lenny_or_Sid,
this should be ready to work.
Also, I have downloaded the latest SVN revision from
http://svn.openstreetmap.org/applications/rendering/mapnik and done
the steps described on
http://wiki.openstreetmap.org/wiki/Mapnik#Rendering_with_Mapnik.
I have imported the extract for the UK
(http://nick.dev.openstreetmap.org/downloads/planet/uk-090701.osm.bz2)
with osm2pgsql. No problem with the import and everything looks good
in the PostGIS db. The following tables are populated with
reasonably looking data:
Schema |Name| Type | Owner
--++--+
public | planet_osm_line| table | hansm
public | planet_osm_nodes | table | hansm
public | planet_osm_point | table | hansm
public | planet_osm_polygon | table | hansm
public | planet_osm_rels| table | hansm
public | planet_osm_roads | table | hansm
public | planet_osm_ways| table | hansm
osm= SELECT * FROM geometry_columns;
f_table_catalog | f_table_schema |f_table_name|
f_geometry_column | coord_dimension | srid |type
-+++---+-++
| public | planet_osm_point |
way | 2 | 900913 | POINT
| public | planet_osm_line|
way | 2 | 900913 | LINESTRING
| public | planet_osm_polygon |
way | 2 | 900913 | POLYGON
| public | planet_osm_roads |
way | 2 | 900913 | LINESTRING
When trying to render tiles with generate_tiles, I only get PNG
files with waterbodies and land, but not the actual OSM features.
This is independend from the zoom level.
When checking my PostgreSQL log files, I see malformed SQL queries
issued by mapnik. Obviously, it does not know about the tables'
geometry columns:
select asbinary() as geom,highway,railway from
(select way,highway,railway,
case when tunnel in ('yes','true','1') then
'yes'::text else tunnel end as tunnel
from planet_osm_roads
where highway is not null
or railway is not null
order by z_order
) as roads
where setSRID('BOX3D(-19724422.27493316
7200979.560689881,-18472078.00350883
8453323.832114212)'::box3d,537534573)
Now my big question: How do I explain mapnik that it has to use
way as geometry column?
Thanks for helping
Hans
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
Am Mittwoch, den 01.07.2009, 09:47 +1000 schrieb Brett Henderson: Jeffrey Warren wrote: And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? I don't mind really. I only make them available as a convenience so people don't need to run the rails migrations themselves. I don't want to put much effort into maintaining them so I just leave them as is. Is this something you'd be interested in looking after? I am somewhat confused, maybe I missed something. Either you can recreate the constraint if the import is done or the data is _corrupt_. If you are loading a larger volume of data you probably want to drop _all_ constraints and indexes before that, recreate some in the middle and the rest afterwards. Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapnik doesn't find geometry columns
Hi, Hans Musil wrote: Sure, exactly this was the problem! That's why I wanted to know how to tell Mapnik what the geometry column is. It didn't find it by itself and just left it blank in the SQL query. I had similar problems in the past, intermittently. They do not stem from Mapnik not finding the geometry columns - it does an initialisation query on startup which you can find in the PostGIS log files if you look closely. However, there seem to be situations where Mapnik (or someone else) drops the PostGIS connection halfway through, and then Mapnik is clever enough to allocate a new connection from its pool but for some reason the initialising query is not always made for all connections and you end up having some stale ones in the pool that lack the geo column info. At least that was my diagnosis at the time... I think my application had somehow messed up proper PostGIS connectivity or so. Look for geometryColumn_=rs-getValue(f_geometry_column); in the Postgis input plugin and put some debug print statements there to find out what is going on. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapnik doesn't find geometry columns
Hans Musil wrote: I'm running mapnik version 0.5.1 as shipped with Debian Lenny. Following http://wiki.openstreetmap.org/wiki/Mapnik/Installation#For_Debian_Lenny_or_Sid, this should be ready to work. You might investigate if it's feasible for you to upgrade. Mapnik 0.6.0 has been out for quite a while now, and 0.6.1 is imminent. the PostGIS db. The following tables are populated with reasonably looking data: I presume you also have the spatial_ref_sys table? | public | planet_osm_line| way | 2 | 900913 | LINESTRING Mapnik 0.5.1 performs this query to find the geometry column, for instance for the line table: select f_geometry_column,srid,type from geometry_columns where f_table_name='planet_osm_line'; Then picks up the f_geometry_column as the name of the geometry. What does that return for you? select asbinary() as geom,highway,railway from Hmm, weird. Why is your mapnik not also selecting tunnel? tunnel is used in the roads style filters. Now my big question: How do I explain mapnik that it has to use way as geometry column? As above: it should pick it up by itself. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
On 7 Jul 2009, at 18:47, Joachim Zobel wrote: Am Mittwoch, den 01.07.2009, 09:47 +1000 schrieb Brett Henderson: Jeffrey Warren wrote: And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? I don't mind really. I only make them available as a convenience so people don't need to run the rails migrations themselves. I don't want to put much effort into maintaining them so I just leave them as is. Is this something you'd be interested in looking after? I am somewhat confused, maybe I missed something. Either you can recreate the constraint if the import is done or the data is _corrupt_. If you are loading a larger volume of data you probably want to drop _all_ constraints and indexes before that, recreate some in the middle and the rest afterwards. Could it be that your constraint failures are due to a node/way/ relation containing two keys that have different capitalisation? Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
No, the data is not corrupt, nor are there duplicate entries. Quite a few current_way_nodes are not getting deleted when they cease to be 'current', that is, once they're only history and no longer part of the current planet. I asked earlier about what mechanism actually deletes current_way_nodes if marked for deletion (i think the tag is 'visible=false' but i might be mistaken) but nobody seemed to know - it seems the best route is to go through and delete those orphans, but I couldn't write a single SQL query to do so efficiently and before writing a script to do it, I realized that the rails port doesn't really care if they're there, or if the constraint is created. Jeff On Tue, Jul 7, 2009 at 3:24 PM, Shaun McDonald sh...@shaunmcdonald.me.ukwrote: On 7 Jul 2009, at 18:47, Joachim Zobel wrote: Am Mittwoch, den 01.07.2009, 09:47 +1000 schrieb Brett Henderson: Jeffrey Warren wrote: And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? I don't mind really. I only make them available as a convenience so people don't need to run the rails migrations themselves. I don't want to put much effort into maintaining them so I just leave them as is. Is this something you'd be interested in looking after? I am somewhat confused, maybe I missed something. Either you can recreate the constraint if the import is done or the data is _corrupt_. If you are loading a larger volume of data you probably want to drop _all_ constraints and indexes before that, recreate some in the middle and the rest afterwards. Could it be that your constraint failures are due to a node/way/ relation containing two keys that have different capitalisation? Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Slow WMS Yahoo (WebKit)
Hi Does anybody noticed slow connection to Yahoo-API-Server? When I use Yahoo-WMS-Layer it takes 2-5 minutes for a single picture to load. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Slow WMS Yahoo (WebKit)
Am 07.07.2009 19:22, Peter Herison: Hi Does anybody noticed slow connection to Yahoo-API-Server? When I use Yahoo-WMS-Layer it takes 2-5 minutes for a single picture to load. Takes around 5-10 seconds per tile here. Did you update your WMS Plugin lately? Make sure that you're using the new Yahoo Sat WMS entry pointing to html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?; Claudius ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Slow WMS Yahoo (WebKit)
On Tue, Jul 7, 2009 at 8:15 PM, Claudiusclaudiu...@gmx.de wrote: Am 07.07.2009 19:22, Peter Herison: Hi Does anybody noticed slow connection to Yahoo-API-Server? When I use Yahoo-WMS-Layer it takes 2-5 minutes for a single picture to load. Takes around 5-10 seconds per tile here. Did you update your WMS Plugin lately? Make sure that you're using the new Yahoo Sat WMS entry pointing to html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?; Why isn't this file installed locally like the old Yahoo HTML wrapper? ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Slow WMS Yahoo (WebKit)
Claudius schrieb: Am 07.07.2009 19:22, Peter Herison: Does anybody noticed slow connection to Yahoo-API-Server? When I use Yahoo-WMS-Layer it takes 2-5 minutes for a single picture to load. Takes around 5-10 seconds per tile here. Did you update your WMS Plugin lately? Make sure that you're using the new Yahoo Sat WMS entry pointing to html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?; Hmmm... I updated WMS-Plugin. Now I'm using JOSM 1741 and WMS-Plugin 16363, but I get the red Error-Pattern with YahooDirect. ---***--- java.lang.Exception: Could not start browser. Please check that the executable path is correct. Cannot run program webkit-image-gtk: CreateProcess error=2, Das System kann die angegebene Datei nicht finden ---***--- I'm using Vista. How do I get this working with webkit-image.exe? %path% is extended to d:\Program Files\josm where josm also webkit-image.exe resides. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
