Re: [OSM-dev] Is there a way to use simple schema without hstore
Hi Brett, thanks for your elaborate answer! Now I am up to date. Some ideas regarding my use case ... Am 18.11.10 23:50, schrieb Brett Henderson: Hi Andreas, The change was made mostly for performance reasons. With a full planet imported into the database, bounding box style queries are now approximately 10 times faster. This is due to a couple of reasons: * All data (with the exception of relations) is now clustered by geographical location. This drastically improves performance where data is being processed for a limited area. * The nodes and ways tables are the only tables that have a geometry column, thus other data must be embedded in those tables in order to make use of clustering. My concept is always to use _1_ table for all geometries and to create extractions when I need them. Because a geom column can store any type, so it is a more unifying concept. I don't understand your comment regarding NoSQL. The main change is that now you will have to deal with a more complex hstore column type on the nodes/ways tables, but otherwise the same data still exists and can still be manipulated with SQL statements. The data is less relational that it was previously, but tag data is not terribly useful without access to parent entities so grouping them together shouldn't result in loss of functionality. You can still populate separate tags tables if you wish by running your own separate query to pull the hstore column apart. This is what I need to do sooner or later, when I will update. It's important for me to use a separate table for tags, because I run a script that will correct the tags of relations (from outer ways to relations), and I don't want to rewrite this and other scripts that depend on this schema every time the version changes. Running an extra script that fetches the hstore tags and puts them into a separate table will add the time that PBF gave me ;( My main concern is that with the next big schema update I _have_ to patch the schema. On the long run it is great to be conservative about such changes or *) If you're applying diffs to the database you can enhance the osmosisUpdate() function (initially empty, but can be customised) to keep your separate tags tables up to date during each diff application. You will need to run the pgsql_simple_schema_0.6_action.sql script against the database so that all actions during a diff are logged and can be used by your osmosisUpdate function to know which records need to be re-processed. Is it possible to truncate the actions table for myself so that a separate script can access the changes? This is another important point. In the moment, I manually populate my own current_features tables after an update that are populated with all features, whose tstamp is = the time of the last update. A little overhead ... I see that this table exists in 0.36 as well, so I could use it, if I can truncate it manually? The older Osmosis 0.36 is still available so you don't have to upgrade. It remains compatible with 0.6 XML files. Finally, if there is enough demand for the older schema style the old tasks can be pulled back out of SVN and run alongside the new ones, but I'm not keen to do that without good reason. I did consider trying to support both styles of table in the same tasks by dynamically detecting what tables are installed, but it increases the code complexity considerably and I didn't think the effort was worthwhile. *) With that, you would provide a downward compatible solution that I would appreciate a lot! Is it necessary that Osmosis makes the schema checks? What about giving each schema a unique ID and then let the user point Osmosis to this ID and let it fail, if the user has installed the wrong schema? Finally, I didn't make the change without careful consideration. I do try to keep schemas stable, and when they do change I provide an upgrade script to allow migration between them. But the performance gains achieved through use of hstore were too great to ignore. Retrieving heavily populated 1x1 degree areas from a database containing a full planet used to take approximately 1 hour, but this is now down to well under 10 minutes. On the long run, this is an argument ;) I am critical, because I still haven't thought through all dependant scripts that do something with tags. But there are many ... Hope that helps, Brett On Thu, Nov 18, 2010 at 8:18 PM, Andreas Kalsch andreaskal...@gmx.de mailto:andreaskal...@gmx.de wrote: Is there a way to use simple schema in Osmosis without hstore? And why was this changed? A separate table for tags can more easily be indexed. I think it is not a good idea to use hstore because then we can drop SQL, use NoSQL for storing data and use PostGIS/Postgres for Geometry only. What do you think? Best, Andi ___ dev mailing list
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
M∡rtin Koppenhoefer dieterdre...@gmail.com wrote: yes, I did this because I don't know how to get a proper package for mapnik, thank you. This should be as simple as apt-get install libmapnik-dev Sven -- Der normale Bürger ist nicht an der TU Dresden und schreibt auch nicht mit mutt. (Ulli Kuhnle in de.comp.os.unix.discussion) /me is gig...@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Andreas Kalsch andreaskal...@gmx.de wrote: I think it is not a good idea to use hstore because then we can drop SQL, use NoSQL for storing data and use PostGIS/Postgres for Geometry only. In the real world there is no black and white! Shure, hstore is comparable to NoSQL aproaches, but why should it be a bad thing to use a best of both worlds aproach? Sven -- Den Rechtsstaat macht aus, dass Unschuldige wieder frei kommen (Wolfgang Schäuble) /me is gig...@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
One simple answer: The drivers do not work appropriately with complex SQL data types. In PHP or node.js I will get a string that I have to parse, in MongoDB, I get a proper object or list. If I used hstore in a consequent way (I like consequence and unification), I would have sets in sets, and this is the same as a document oriented database. But just intermingling things for fun does not make the world better. MongoDB, for example, unifies worlds by simply using JSON. I don't have to manually parse things I do not need to parse. Am 19.11.10 09:47, schrieb Sven Geggus: Andreas Kalschandreaskal...@gmx.de wrote: I think it is not a good idea to use hstore because then we can drop SQL, use NoSQL for storing data and use PostGIS/Postgres for Geometry only. In the real world there is no black and white! Shure, hstore is comparable to NoSQL aproaches, but why should it be a bad thing to use a best of both worlds aproach? Sven ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Hi, Andreas Kalsch wrote: One simple answer: The drivers do not work appropriately with complex SQL data types. In PHP or node.js I will get a string that I have to parse, in MongoDB, I get a proper object or list. If I used hstore in a consequent way (I like consequence and unification), I would have sets in sets, It seems to me that you are mistaking consequence for exaggeration. In many cases - especially when dealing with large real-world datasets as opposed to a nice little hello-world application -, a healthy compromise works better than grabbing one concept and trying to make the world fit that concept. But just intermingling things for fun does not make the world better. I think you're misunderstanding. hstore has not been implemented for fun. (Are you aware that PostgreSQL can extend column indexes to hstore keys?) MongoDB, for example, unifies worlds by simply using JSON. I don't have to manually parse things I do not need to parse. In turn, you will have a hard time getting the performance required for a planet-wide application out of MongoDB. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Am 19.11.10 10:06, schrieb Frederik Ramm: Hi, Andreas Kalsch wrote: One simple answer: The drivers do not work appropriately with complex SQL data types. In PHP or node.js I will get a string that I have to parse, in MongoDB, I get a proper object or list. If I used hstore in a consequent way (I like consequence and unification), I would have sets in sets, It seems to me that you are mistaking consequence for exaggeration. In many cases - especially when dealing with large real-world datasets as opposed to a nice little hello-world application -, a healthy compromise works better than grabbing one concept and trying to make the world fit that concept. I am sure there are some good uses for hstore, but as soon as you use it, you are waiting for something like a document-oriented database. I ask myself: Why do I need normal columns when there is hstore? Of course there are some answers like special indexing ... the fact: Intermingling both concepts inside one database will make queries and schema design more complex than necessary - many, many time-consuming choices you do not need to do in the NoSQL world. If you take a look at all Postgres data types, you have a myriad of choices. Often, a simple design will win, especially when you will build something more complex on top of it. It's only one step away from switching to a document store. Example for unnessessary complex schema design: http://wiki.openstreetmap.org/wiki/DE:HowTo_minutely_hstore My personal point is that my system relies on the 0.36 schema and I simply cannot change all dependant scripts. But just intermingling things for fun does not make the world better. I think you're misunderstanding. hstore has not been implemented for fun. (Are you aware that PostgreSQL can extend column indexes to hstore keys?) Probably I am wrong ... yes I know that you can index hstore with a GIST. MongoDB, for example, unifies worlds by simply using JSON. I don't have to manually parse things I do not need to parse. In turn, you will have a hard time getting the performance required for a planet-wide application out of MongoDB. OK, can you explain further what the bottlenecks would be? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Minutely Mapnik
Hi, On 18.11.2010 22:24, Frederik Ramm wrote: Martijn van Exel wrote: Does the process of applying minutely updates only work when the initial planet was a full planet, not an extract? Is it possible to do this on extracts? Yes (use the bounding box filter on osm2pgsql), but there will be some fuzziness around the edges. I forgot what exactly it was, but there will be some cases where a node is moved from the inside to the outside or so where things foul up. This is one point. It can be solved by using a slightly larger bounding box than required. More annoying is that the ways and relation tables grow. This is due to missing checks for their geometry against the bounding box. I still believe osm2pgsql can be extended to handle this. It will slow down the import as it needs to do additional checks. I was too lazy to investigate this further. I run a cleanup from time to time to do the job. I have the feeling the cleanup uses less resources than the overhead generated by checking the geometries on every update. You can find the cleanup queries here: http://wiki.openstreetmap.org/wiki/User:Stephankn/knowledgebase#Cleanup_of_ways_outside_the_bounding_box Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Removing scale_max / scale_min from elemstyles.xml? / mapcss
Ulf Lamping wrote: Am 15.11.2010 23:48, schrieb Sebastian Klein: No objections. Maybe we can switch the default value of mappaint.zoomLevelDisplay from false to true afterwards. This way authors of external style sets can create fancy scale dependent styles and don't need to bother their users with advanced preferences configuration. Makes sense. To avoid any hassle, we may want to remove the scale entries first, wait one or two releases and then change the default value. Generally I don't see any problem since the xml style is distributed with the binary. But users who have loaded a modified copy of elemstyles.xml would be affected. Btw., I have mostly completed mapcss support (which offers zoom dependent markup, as well), but it will take some time to integrate into JOSM. A lot of existing code needs to be adapted and I don't want to break too much... E.g. the z-index property does not work well with the way multipolygons are drawn in the PaintVisitor. Do you keep an eye on the performance? Having a new rendering engine that is able to display a lot of fancy stuff but is also a lot slower might be the wrong direction ... ;-) Yes, I'm aware of this. Currently, the new PaintVisitor is slightly slower (basically due to z-level ordering) but not significantly ( 5% I think, but I will do careful benchmarking). All styles are cached, of course, so most time is not spend in JOSM code, but with actual rendering of areas and lines. Regards, ULFL P.S: I'm not a big fan of mapcss, as the XML we currently have makes it a lot easier for externals to automatically analyze the rendering (and preset) rules files (as e.g. Jochen is currently doing it for taginfo). Parsing the css alike syntax isn't impossible, but obviously harder compared to XML that a lot of people already know how to do - including myself ;-) Sure, it's not that easy to parse, but it's a matter of priority. E.g. you don't design a programming language such that compiler has an easy job to parse the source code, but from a user perspective. (Otherwise Lisp would be much more popular and Perl much less. :) ) Another reason for supporting this format is its use in potlatch 2, so we might get a second style in JOSM for free. Maintaining a single style must be a lot of work already, so why not join forces a little. I'm not about to remove the xml style support, though. It will still be available and you can even load style files with different formats at the same time. (But as I said, don't expect it before Christmas. :) ) Sebastian ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Andreas Kalsch andreaskal...@gmx.de wrote: Example for unnessessary complex schema design: http://wiki.openstreetmap.org/wiki/DE:HowTo_minutely_hstore You are welcome to design a better database scheme suitable for rendering :) osm2pgsql output is evolution _not_ design. Using a join in every single SQL request is not an Option in this case at least performancewise. Probably I am wrong ... yes I know that you can index hstore with a GIST. GIN Sven -- Threading is a performance hack. (The Art of Unix Programming by Eric S. Raymond) /me is gig...@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tool to convert OSM ways into multi-point segments?
I'm wanting to add a feature to Freemap (UK OSM-based hiking site) to allow users to tag individual trails with route descriptions, interesting features etc - with the eventual aim of allowing users to select a walking route and have a ready-made route description (note these descriptions will be stored in my own database... no need to worry about overloading OSM with this!) However, to do this effectively I need to split ways at junctions, e.g. if a trail has 3 junctions it should be split into 4 multi-point segments, because a walking route will consist of several of these segments in a line or loop. (The more I think about it, the more I think that maybe we should have kept segments - the only problem with them was that they were restricted to 2 points -but anyway) Anyway the crux of the matter is I'm wondering if anyone has a tool to do this already (to either write out an OSM file or write to a database) I could easily do one myself but obviously if there's something out there already I may as well use it. Thanks, Nick ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Hi, On 11/19/10 10:25, Andreas Kalsch wrote: I am sure there are some good uses for hstore, but as soon as you use it, you are waiting for something like a document-oriented database. I ask myself: Why do I need normal columns when there is hstore? This is exactly what I was getting at. If you're driven by a desire for the pure, then you'd drop all the extra columns in the tables and have tags *only* in hstore. But it turns out that even if you go through the effort of adapting your mapnik style sheet to query the hstore columns, this doesn't perform well enough for proper tile rendering, so you re-add those columns that are used often... MongoDB, for example, unifies worlds by simply using JSON. I don't have to manually parse things I do not need to parse. In turn, you will have a hard time getting the performance required for a planet-wide application out of MongoDB. OK, can you explain further what the bottlenecks would be? I've never touched MongoDB, I've just heard people enthusiastically adopt MongoDB for OSM and then growing very quiet. I think you were involved in that thread (back in July)? But to go back to your original problem - I have a similar situation where a plugin has been written for Osmosis 0.33 which doesn't work with 0.37 any more, and because I could not be bothered to fix that, I simply use two instances of Osmosis in a pipe - one 0.37 instance to handle the pbf and one 0.33 instance to interface with my plugin. Ugly but works. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
I agree - my approach is the playground approach: At first, keep it simple. For your purpose it is correct to split the table. What about a partial index over 1 table? This is possible with Postgres. Am 19.11.10 11:03, schrieb Sven Geggus: Andreas Kalschandreaskal...@gmx.de wrote: Example for unnessessary complex schema design: http://wiki.openstreetmap.org/wiki/DE:HowTo_minutely_hstore You are welcome to design a better database scheme suitable for rendering :) osm2pgsql output is evolution _not_ design. Using a join in every single SQL request is not an Option in this case at least performancewise. Probably I am wrong ... yes I know that you can index hstore with a GIST. GIN Sven ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
On Fri, Nov 19, 2010 at 09:37:33AM +0100, Andreas Kalsch wrote: If you're applying diffs to the database you can enhance the osmosisUpdate() function (initially empty, but can be customised) to keep your separate tags tables up to date during each diff application. You will need to run the pgsql_simple_schema_0.6_action.sql script against the database so that all actions during a diff are logged and can be used by your osmosisUpdate function to know which records need to be re-processed. Is it possible to truncate the actions table for myself so that a separate script can access the changes? Simply copy away the information from the action table somewhere persistent in the osmosisUpdate function. Works fine. However, +1 from me for an action table that can be truncated manually. The older Osmosis 0.36 is still available so you don't have to upgrade. It remains compatible with 0.6 XML files. Finally, if there is enough demand for the older schema style the old tasks can be pulled back out of SVN and run alongside the new ones, but I'm not keen to do that without good reason. I did consider trying to support both styles of table in the same tasks by dynamically detecting what tables are installed, but it increases the code complexity considerably and I didn't think the effort was worthwhile. *) With that, you would provide a downward compatible solution that I would appreciate a lot! I haven't tried yet but it should be fairly simple to create a view that simulates the old tag tables. Probably not the best solution performance- wise but useful for a gradual update of old scripts. Sarah ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tool to convert OSM ways into multi-point segments?
Nick, I am sure that all routing programs must do something similar, but last year Hermann Kraus produced srtm2wayinfohttp://wiki.openstreetmap.org/wiki/Srtm2wayinfofor his Google Summer of Code Project. It does what you are talking about - looks for junctions between ways and splits the ways into sections between junctions. The output is a set of relations representing each section. The 'srtm' part of the program is that it will also give you the total climb in each section too. Graham. On 19 November 2010 10:04, Nick Whitelegg nick.whitel...@solent.ac.ukwrote: I'm wanting to add a feature to Freemap (UK OSM-based hiking site) to allow users to tag individual trails with route descriptions, interesting features etc - with the eventual aim of allowing users to select a walking route and have a ready-made route description (note these descriptions will be stored in my own database... no need to worry about overloading OSM with this!) However, to do this effectively I need to split ways at junctions, e.g. if a trail has 3 junctions it should be split into 4 multi-point segments, because a walking route will consist of several of these segments in a line or loop. (The more I think about it, the more I think that maybe we should have kept segments - the only problem with them was that they were restricted to 2 points -but anyway) Anyway the crux of the matter is I'm wondering if anyone has a tool to do this already (to either write out an OSM file or write to a database) I could easily do one myself but obviously if there's something out there already I may as well use it. Thanks, Nick ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Graham Jones Hartlepool, UK. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapnik / General question on projection
Andy Allan kirjoitti: On Thu, Nov 18, 2010 at 5:58 PM, Mâ¡rtin Koppenhoefer dieterdre...@gmail.com wrote: First I must admit that I don't know much about projections, reference systems and geodetic datums. I have recently experimented with homemade hillshading and contours, and I found out by trying that my hillshade was fitting the contours if I assumed it was in EPSG 3785 (I created the hillshading from SRTM with Grass GIS and reprojected to an EPSG 900913 setting while the contours were created with mapnik and srtm2osm). By reading the EPSG documentation sheets it seems that 3785 and 3857 are not the same (the first uses a sphere, while the latter uses an ellipsoid with inverse flattening of 298.257223563). EPSG says that the ellipsoid-version is valid (and registered as 3857) while the sphere is not. This results visually in quite some change on the map if you mix it up. Therefor I thought that Mapnik's 900913 was equal to EPSG's non-valid 3785 version with the sphere. (EPSG states: Sphere with radius equal to the semi-major axxis of the GRS80 and WGS84 ellipsoids. Used only for Web approximate mapping and visualisation. Not recognised by geodetic authorities --- on which I found some comments (EPSG is arrogant) in the web btw. who said it was perfectly working with the sphere as well, even if not scientifically correct). Now I found some docs in the OSM-Wiki that state we were using EPSG 3857. My own tests indicate we are using the (not registered) 3785 (sphere). Also this setting in mapnik/inc/entities.xml.inc indicates this: !ENTITY srs900913 +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +no_defs +over in generate_image.py I found this (seems that someone changed from sphere to ellipsoid): Â Â # prj = mapnik.Projection(+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +no_defs +over) Â Â prj = mapnik.Projection(+proj=merc +a=6378137 +rf=298.257223563 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +no_defs +over) can someone please tell me which datum is used by the mapnik and t...@h tile servers? I'm totally confused by your email, so bear with me. As far as I'm aware, 900913, 3785 and 3857 are all identical, and don't have the difference that you are describing above. From spatialreference.org http://www.spatialreference.org/ref/epsg/3785/ http://www.spatialreference.org/ref/epsg/3785/proj4/ http://spatialreference.org/ref/sr-org/6864/ (EPSG:3857) http://spatialreference.org/ref/sr-org/6864/proj4/ From the proj4 project: http://trac.osgeo.org/proj/browser/trunk/proj/nad/epsg Moreover, you refer to changes made to the generate_image.py file, but I don't see such a change: http://trac.openstreetmap.org/browser/applications/rendering/mapnik/generate_image.py#L38 However, the following two entries at EPSG are conflicting with all the above, and do in fact appear to be different: http://www.epsg-registry.org/report.htm?type=selectionentity=urn:ogc:def:crs:EPSG::3785reportDetail=shortstyle=urn:uuid:report-style:default-with-codestyle_name=OGP%20Default%20With%20Codetitle=EPSG:3785 http://www.epsg-registry.org/report.htm?type=selectionentity=urn:ogc:def:crs:EPSG::3857reportDetail=shortstyle=urn:uuid:report-style:default-with-codestyle_name=OGP%20Default%20With%20Codetitle=EPSG:3857 Ho hum. Maybe take it up with the proj project? Anyway, we use Spherical Mercator ie. the one without the +rf=298.25... so I'd find a different copy of generate_image.py Hi, I asked this from gdal-dev list and this is what Frank Warmerdam wrote: Jukka, In EPSG:3785 web mercator was expressed as a normal Mercator 1SP projection on a spherical GCS. However, this lost the fact that the lat/long coordinates you produced when unprojecting the mercator coordiantes were supposed to be treated as WGS84, not lat/long on the sphere. So that was deprecated and EPSG:3857 (it freaks me out that they are the same digits just rearranged!) was introduced. This is based on WGS84, but they define a custom projection method Popular Visualization Psuedo Mercator which seems to have the spherical earth model for the projection algorithm baked in, but is otherwise based on a normal WGS84 GCS. This makes it easy for applications to understand that the unprojected coordinates are to be treated as WGS84. For what it's worth, GDAL has custom logic to treat them as equivalent. Unfortunately I do not really understand the answer, but I suppose that if gdal is recognising EPSG:3857 then it treats it right. -Jukka Rahkonen- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
On Fri, Nov 19, 2010 at 7:37 PM, Andreas Kalsch andreaskal...@gmx.dewrote: Hi Brett, thanks for your elaborate answer! Now I am up to date. Some ideas regarding my use case ... Am 18.11.10 23:50, schrieb Brett Henderson: Hi Andreas, The change was made mostly for performance reasons. With a full planet imported into the database, bounding box style queries are now approximately 10 times faster. This is due to a couple of reasons: - All data (with the exception of relations) is now clustered by geographical location. This drastically improves performance where data is being processed for a limited area. - The nodes and ways tables are the only tables that have a geometry column, thus other data must be embedded in those tables in order to make use of clustering. My concept is always to use _1_ table for all geometries and to create extractions when I need them. Because a geom column can store any type, so it is a more unifying concept. I'm not following. Are you suggesting that node and way tables be merged? I don't understand your comment regarding NoSQL. The main change is that now you will have to deal with a more complex hstore column type on the nodes/ways tables, but otherwise the same data still exists and can still be manipulated with SQL statements. The data is less relational that it was previously, but tag data is not terribly useful without access to parent entities so grouping them together shouldn't result in loss of functionality. You can still populate separate tags tables if you wish by running your own separate query to pull the hstore column apart. This is what I need to do sooner or later, when I will update. It's important for me to use a separate table for tags, because I run a script that will correct the tags of relations (from outer ways to relations), and I don't want to rewrite this and other scripts that depend on this schema every time the version changes. Running an extra script that fetches the hstore tags and puts them into a separate table will add the time that PBF gave me ;( My main concern is that with the next big schema update I _have_ to patch the schema. On the long run it is great to be conservative about such changes or *) My best suggestion is to continue running the old Osmosis. The old version still works so don't upgrade. As Frederik suggests you can run the two versions alongside each other and pipe data between them as necessary. If you're applying diffs to the database you can enhance the osmosisUpdate() function (initially empty, but can be customised) to keep your separate tags tables up to date during each diff application. You will need to run the pgsql_simple_schema_0.6_action.sql script against the database so that all actions during a diff are logged and can be used by your osmosisUpdate function to know which records need to be re-processed. Is it possible to truncate the actions table for myself so that a separate script can access the changes? This is another important point. In the moment, I manually populate my own current_features tables after an update that are populated with all features, whose tstamp is = the time of the last update. A little overhead ... I see that this table exists in 0.36 as well, so I could use it, if I can truncate it manually? As Sarah suggests, the way to do this is to create your own table and populate it from the actions table within the osmosisUpdate function. The overhead in doing this should be relatively small. The older Osmosis 0.36 is still available so you don't have to upgrade. It remains compatible with 0.6 XML files. Finally, if there is enough demand for the older schema style the old tasks can be pulled back out of SVN and run alongside the new ones, but I'm not keen to do that without good reason. I did consider trying to support both styles of table in the same tasks by dynamically detecting what tables are installed, but it increases the code complexity considerably and I didn't think the effort was worthwhile. *) With that, you would provide a downward compatible solution that I would appreciate a lot! I'm hesitant to do this for one person's use case. I don't mean to be unhelpful, but I have to be very careful about where spend my limited time on Osmosis and for this reason I try to keep things as simple as possible. That sometimes means removing old obsolete functionality and breaking compatibility from time to time. Can you make do with the older version for now? If enough demand comes up for the old style schema then I'll look into it. Alternatively I'm always happy to accept a patch :-) Is it necessary that Osmosis makes the schema checks? What about giving each schema a unique ID and then let the user point Osmosis to this ID and let it fail, if the user has installed the wrong schema? I don't understand your suggestion. The schema currently has a schema_info
Re: [OSM-dev] Is there a way to use simple schema without hstore
On Fri, Nov 19, 2010 at 8:25 PM, Andreas Kalsch andreaskal...@gmx.dewrote: Am 19.11.10 10:06, schrieb Frederik Ramm: Hi, Andreas Kalsch wrote: One simple answer: The drivers do not work appropriately with complex SQL data types. In PHP or node.js I will get a string that I have to parse, in MongoDB, I get a proper object or list. If I used hstore in a consequent way (I like consequence and unification), I would have sets in sets, It seems to me that you are mistaking consequence for exaggeration. In many cases - especially when dealing with large real-world datasets as opposed to a nice little hello-world application -, a healthy compromise works better than grabbing one concept and trying to make the world fit that concept. I am sure there are some good uses for hstore, but as soon as you use it, you are waiting for something like a document-oriented database. I ask myself: Why do I need normal columns when there is hstore? Of course there are some answers like special indexing ... the fact: Intermingling both concepts inside one database will make queries and schema design more complex than necessary - many, many time-consuming choices you do not need to do in the NoSQL world. If you take a look at all Postgres data types, you have a myriad of choices. Often, a simple design will win, especially when you will build something more complex on top of it. It's only one step away from switching to a document store. Example for unnessessary complex schema design: http://wiki.openstreetmap.org/wiki/DE:HowTo_minutely_hstore Please keep in mind that the one and only reason I've switched to hstore is performance. It has nothing to do with any perceived improvements in schema design or adherence to an alternative data storage philosophy. It most certainly wasn't done for fun ;-) I only switched after spending many days trying alternate ways of indexing the database, waiting (in some cases for several days) for full index builds to occur, and re-running benchmarks to measure improvements. It was an incredibly tedious and frustrating experience that I only continued with in order to make the database scale more effectively to planet sized datasets. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
On Fri, Nov 19, 2010 at 10:50 PM, Sarah Hoffmann lon...@denofr.de wrote: On Fri, Nov 19, 2010 at 09:37:33AM +0100, Andreas Kalsch wrote: If you're applying diffs to the database you can enhance the osmosisUpdate() function (initially empty, but can be customised) to keep your separate tags tables up to date during each diff application. You will need to run the pgsql_simple_schema_0.6_action.sql script against the database so that all actions during a diff are logged and can be used by your osmosisUpdate function to know which records need to be re-processed. Is it possible to truncate the actions table for myself so that a separate script can access the changes? Simply copy away the information from the action table somewhere persistent in the osmosisUpdate function. Works fine. However, +1 from me for an action table that can be truncated manually. Is there likely to be a noticeable performance improvement in doing this? My preference if the overhead is small would be to add a contrib script to Osmosis that installs a non-truncating table that is updated by osmosisUpdate, and a customised osmosisUpdate function. It keeps the pgsql tasks simpler if I can do that. Then again, this is probably not a complicated change to make to Osmosis itself. I'd need to take a look ... Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Hmm, I've given all of this a bit more thought. Perhaps there is a need for a simple schema that is easy for people to populate and utilise. I'm quite happy with hstore, but it's not as simple for those familiar with basic SQL. The original reason I created the so-called simple schema was to support improved bounding box functionality because I couldn't do it via flat files efficiently. It was called simple because I was also working on a full history schema that I never found time to complete. My intent has always been to optimise for accurate bounding box query performance and not simplicity so the name is something of a misnomer. Anyway, perhaps I should re-instate the old tasks and run them alongside the new ones. I'll have to re-think the naming of these tasks and schemas. Perhaps simple and snapshot or something ... But I don't know when I'll get to do this. I'm very time poor at the moment. On Sat, Nov 20, 2010 at 11:47 AM, Brett Henderson br...@bretth.com wrote: Please keep in mind that the one and only reason I've switched to hstore is performance. It has nothing to do with any perceived improvements in schema design or adherence to an alternative data storage philosophy. It most certainly wasn't done for fun ;-) I only switched after spending many days trying alternate ways of indexing the database, waiting (in some cases for several days) for full index builds to occur, and re-running benchmarks to measure improvements. It was an incredibly tedious and frustrating experience that I only continued with in order to make the database scale more effectively to planet sized datasets. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
On Sat, Nov 20, 2010 at 12:09 PM, Brett Henderson br...@bretth.com wrote: Hmm, I've given all of this a bit more thought. Perhaps there is a need for a simple schema that is easy for people to populate and utilise. I'm quite happy with hstore, but it's not as simple for those familiar with basic SQL. The original reason I created the so-called simple schema was to support improved bounding box functionality because I couldn't do it via flat files efficiently. It was called simple because I was also working on a full history schema that I never found time to complete. My intent has always been to optimise for accurate bounding box query performance and not simplicity so the name is something of a misnomer. Anyway, perhaps I should re-instate the old tasks and run them alongside the new ones. I'll have to re-think the naming of these tasks and schemas. Perhaps simple and snapshot or something ... But I don't know when I'll get to do this. I'm very time poor at the moment. Okay, that wasn't too painful. I found an hour or two to today so I've created new (actually old) tasks and documented them here: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39#PostGIS_Tasks_.28Simple_Schema.29 It's available in SVN for now. These new tasks are just the old tasks retrieved out of SVN history and renamed. Note that these tasks are effectively at a dead end because any new development is likely to proceed on the new hstore schema. They should continue to work for the foreseeable future. I'll probably cause further confusion with schema names. But I'm now calling the new style hstore schema the pgsnapshot schema, and the older tag table style schema the pgsimple schema. I'm open to suggestions, but any change will need to be reflected on the wiki and in the codebase. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[osmosis-dev] Osmosis 0.38 Released
Hi All, I've just released Osmosis 0.38. http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.38.tgz http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.38.zip http://dev.openstreetmap.de:23457/hudson/job/osmosis-RELEASE-ant/38/ From changes.txt: - Updated internal storage to use long instead of int identifiers to remove 2^31 node id limit. - Updated apidb support to the latest schema 20100910084426. - Fixed the --write-pbf short option --wb to point to the writer implementation instead of the reader. - Added additional quoting to commands in the osmosis.bat launcher to fix issues with spaces in the installation path. - Modified plugin loader code to support multiple osmosis-plugins.conf files to allow each plugin to provide its own. Let me know if you see any issues. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [josm-dev] AddPrimitivesCommand constructor
Roman Byshko wrote: Hi guys, currently AddPrimitivesCommand does not have a constructor where OsmLayer can be give as a parameter. I'm developing a plugin right now and need this option. May I ask what the plugin if for? The patch in the attachment adds this functionality. Would you please commit it into the trunk? Done. (in [3660]) Regards, Roman p.s. sorry if the path in the patch is wrong... if so, how it should be? Had no problems with the path, but it is preferred to put patches in a Trac ticket with [PATCH] topic of the patch as title. This way we can discuss the patch without spamming the mailing list and, more importantly, we won't forget about it. Sebastian ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] TIFF support
Olivier Croquette wrote:
Hi !
When I call this code from JOSM :
String[] formats = ImageIO.getReaderFormatNames();
for (String f:formats) {
System.out.println(f);
}
It outputs tiff (lowercase, other variants like tif, TIF and TIFF are
missing).
When I run the same code a simple-hello-world-like program, it doesn't.
I tried the -verbose option of the Java VM, and it gives me :
[Loaded com.sun.imageio.plugins.tiff.TIFFImageReaderSpi from
/Library/Java/JavaVirtualMachines/1.6.0_22-b04-307.jdk/Contents/Classes/classes.jar]
The question is : what in JOSM causes the loading of this class ?
Google sais this class comes from Java Advanced Imaging (JAI) which is
loaded by the ImportImagePlugin. Hope this helps...
Sebastian
___
josm-dev mailing list
josm-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev
