Re: [OSM-dev] OSM Carto stylesheet vs Release osm2pgsql 1.3.0
Thank you for the clarification. Off to upgrade!
Lynn (D)
On 7/29/2020 9:10 AM, Sarah Hoffmann wrote:
On Wed, Jul 29, 2020 at 02:59:24PM +0200, Christoph Hormann wrote:
On Wednesday 29 July 2020, Lynn W. Deffenbaugh (Mr) wrote:
* The pgsql output now looks for lua script relative to the
|style.json| file.
This is a breaking change. Users might have to change the file
names of
their lua scripts in the style files.
Does anyone know if changes need to be made to the OpenStreetMap
Carto stylesheet or is it already compatible with this breaking
change?
OSM-Carto has no style.json file, you specify the lua script directly at
the command line - i think the quoted change applies to the multi/flex
backends only, though the documentation is not very clear in the
terminology used ('pgsql output' vs. 'pgsql backend').
Indeed, that was a minor error in the release notes. I fixed that.
Filenames given on the command line are relative to the current
path. Only file names that appear in style files are interpreted
relative to the style file. The pgsql output used by Carto does
not have the latter. So: nothing changes.
Sarah
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM Carto stylesheet vs Release osm2pgsql 1.3.0
On 7/28/2020 4:39 AM, Sarah Hoffmann wrote: Further changes: * The pgsql output now looks for lua script relative to the |style.json| file. This is a breaking change. Users might have to change the file names of their lua scripts in the style files. Does anyone know if changes need to be made to the OpenStreetMap Carto stylesheet or is it already compatible with this breaking change? I confess that I run a planet-wide tile server with the OSM Carto stylesheet and using osm2pgsql to apply updates but I have NO idea just what is in the stylesheet and how it all works with osm2pgsql. I've put in the documented commands and it "just works", at least pending this question. Lynn (D) ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] mod_tile/renderd serving metatiles
Greetings OSM developers, Has anyone considered, designed, and/or implemented an extension to mod_tile to support serving raw metatile files to clients? I've got my own tile server running using mod_tile and renderd and my own application that consumes OSM-compatible tiles. I really don't like seeing the individual tiles arrive on my Android devices and would like to extend the knowledge of the metatile format into my client application. That way, a single round-trip to the server brings back multiple tiles rather than one per round-trip. The client caches tiles in an MBtiles database, so it can easily extract the individual tiles from the metatile and locally access them as needed. I've looked at mod_tile and it appears to be an easy thing to add. I'm considering extending the /status and /dirty approach to include a /meta option. I'm not completely sold on this because I'm actually planning to return the entire meta tile for the specified /z/x/y.png that was requested along with http response headers describing the metatile parameters. The latter part allows the client to not have to assume that the server is doing 8x8 metatiles but will be told on each response how many are in the file. By returning a full metatile when a single tile is requested (with the /meta trailer), the client doesn't need to be aware of the metatile path naming, but only the format of extracting specific tiles from the metatile file. I'm open to prior art in this area and/or comments on the thinking described above. If I do this, is it something that anyone would be interested in having commited back to the repository? In any case, it will be 100% backward compatible and possibly even a compile or runtime option for mod_tile to include this feature. Lynn (D) - KJ4ERJ - Author of APRSISMO, an Amateur Radio APRS Client for Android under MOAI ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minutely updates format error?
For the past 8-12 (maybe 24) hours, my tile server has been failing to update. The error reported is: Processing: Node(0k 0.0k/s) Cache(Ch:0.0%@0.0%u fh:0.0%) Way(0k 0.00k/s) Relation(0 0.00/s) Why:NodeStart Processing: Node(360k 2.0k/s) Cache(Ch:0.0%@2.9%u fh:98.9%) Way(0k 0.00k/s) Relation(0 0.00/s) Entity: line 395041: parser error : Input is not proper UTF-8, indicate encoding ! Bytes: 0xD8 0x51 0x72 0x20 tag k=name v=Ù¾Ø²Ø´Ú©Û ÙاÙÙÙÛ Ø§Ø³ØªØ§Ù Ø®Ø±Ø§Ø³Ø§Ù ØQr 483 ^ Entity: line 395041: parser error : Unescaped '' not allowed in attributes values tag k=name v=Ù¾Ø²Ø´Ú©Û ÙاÙÙÙÛ Ø§Ø³ØªØ§Ù Ø®Ø±Ø§Ø³Ø§Ù ØQr 483 ^ Entity: line 395041: parser error : attributes construct error tag k=name v=Ù¾Ø²Ø´Ú©Û ÙاÙÙÙÛ Ø§Ø³ØªØ§Ù Ø®Ø±Ø§Ø³Ø§Ù ØQr 483 ^ Entity: line 395041: parser error : Couldn't find end of Start Tag tag tag k=name v=Ù¾Ø²Ø´Ú©Û ÙاÙÙÙÛ Ø§Ø³ØªØ§Ù Ø®Ø±Ø§Ø³Ø§Ù ØQr 483 ^ /var/lib/mod_tile/changes.osc.gz : failed to parse Is there an issue in the data from the database or where might this be coming from? My state.txt file is: #Fri Aug 01 20:03:01 EDT 2014 sequenceNumber=985291 timestamp=2014-08-02T00\:01\:02Z Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] glitch in minutely replication stream
Can you provide cookbook instructions for those of us running OSM tile servers based on the switch2osm.org instructions? Specifically, what does one do to forced them back to process everything from '037+ again? Lynn (D) On 4/18/2014 12:50 PM, Jon Burgess wrote: The minutely replication stream was blocked for 12 hours due to a server problem. The broken sequence number was 834038, with 038.state.txt and 038.osc.gz being zero length files. When the replication was restarted these files were replaced with real data. Unfortunately this has caused a problem for using osmosis to fetch these changes automatically. Once the replication resumed it appears that osmosis skipped the changes in the new '038 files and downloaded '039 onwards. This affected the OSM rendering servers (Yevaud Orm) and I have forced them back to process everything from '037+ again to ensure the changes in the 038 file are rendered. Anyone else running services using the minutely diffs might need to do the same. Jon ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] parameterization of mapnik style-sheets in mod_tile / renderd and multi-lingual maps
Could this feature be used to pass a font-scaling value from the URL to mod_tile assuming that such a scaling value can be used to affect the font sizes rendered in the resulting tile? I'm thinking like .../style/2/z/x/... for instance to double the size of the fonts. This would be very handy to provide for rendering tiles for higher-dpi devices and keeping them readable. Also, are the resulting rendered meta-tiles automatically segregated in the mapnik/apache file store? I sure hope so or there'll be a serious mess of mixed-rendered tiles all under a tree like default. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 10/14/2013 12:40 PM, Kai Krueger wrote: Hello everyone, I would like to mention that I have committed a new feature to mod_tile / renderd and would appreciate any feedback or comments. Mod_tile and renderd are now prepared for the possibility to parameterize the mapnik style-sheet on the fly on a (meta)tile by (metatile) basis. For this purpose, the URL schema for tileservers was extended. It now takes the form: https://my.tile.server/style/parameters/z/x/y.ext parameters is an arbitrary string that gets passed on to renderd and can be used to parameterize the style-sheet for that rendering request. The first use for this functionality is to make it easy to offer multi-lingual maps. I.e. to be able to specify (parameterize) the language in which the names should appear on the map. E.g. one can specify that English names (name:en) should be used instead of the plain name tag. This is heavily based upon Jochen Topf's work on the multi-lingual project for wikimedia [1,2], that has shown the feasibility of this concept. By integrating it into mod_tile / renderd and making it trivial to enable, it will hopefully see wider adoption and prove useful to a bigger audience. You can specify an ordering of name tags to be used when trying to render any labels. E.g. de,en,_ will use name:de if available, otherwise try name:en and finally use name if neither name:de or name:en is available. Technically, renderd will take the stylesheet it loads and replace name in the sql queries with coalesce(tags-'name:de', tags-'name:en', tags-'name') as name. Everything should be backwards compatible and one can activate this feature on a style-by-style basis. If not activated, the old URL schema continues to be used. So e.g. a server can support https://my.tile.server/osm/0/0/0.png and https://my.tile.server/osm-multilingua/de,en,_/0/0/0.png. To make this possible, the protocol between mod_tile and renderd was extended to include the parameterization string (as well as pass through mime information for future purposes, e.g. to be able to render both png and jpg). However, again, everything (both mod_tile and renderd) should be backwards compatible. I.e. renderd can receive rendering requests with the previous version of the protocol, in case you only want to update renderd and still have an older version of mod_tile (or indeed another server software or utility for issuing render requests). Likewise, mod_tile can still send the requests in the previous protocol, and indeed will do so by default unless the parameterization feature is explicitly activated for the given style. This should ensure compatibility with tirex, that has not yet been updated to the change in protocol. Although its first use is for the multi-lingual maps, hopefully this feature can be used for other purposes as well. You still need to write code to interpret the parameterization string and apply the transformation to the mapnik stylesheet object, but given that is all encapsulated in a separate file, this will hopefully be relatively easy if one needs it. Kai [1] http://blog.jochentopf.com/2012-06-21-wikipedia-multilingual-maps-project.html [2] http://blog.jochentopf.com/2012-12-19-status-of-the-multilingual-maps-project.html ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Server configuration: (Urgent problem) pink tiles while requesting lvl 16, 17 18 tiles
And search your syslog for any renderd entries that might provide other clues. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 10/14/2013 1:02 PM, Yves wrote: If you can have a look in the tile server (apache ?) error log, you'll probably find some clues. Look at /var/log/apache2/error.log Yves Andy Allan gravityst...@gmail.com a écrit : On 14 October 2013 17:20, Álvaro Enríquez de Luna Muñoz aenriquezdel...@many-worlds.es wrote: This co-worker isn't working with us anymore, and I am currently facing an urgent problem. Requested tiles from levels 16, 17 and 18 from zones that have never been visited (the app was being used in very restricted areas and now different zones have been enabled) are showing as pink squares. Hi there, It sounds to me like your rendering daemon has died - either renderd, or tirex, depending on your original setup. Without the rendering daemon mod_tile will continue to serve tiles that it has rendered already, but cannot request rendering of any new tiles. The default appearance of missing tiles on OpenLayers is a (nasty) pink background, which again sounds like it fits your situation. Cheers, Andy dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev -- Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Switching tile server to CartoCSS
On 8/28/2013 11:48 AM, Eugene Alvin Villar wrote: Now that the style sheet has been ported to CartoCSS, I would expect improvements to be done. I should have asked this a while ago, but is there any guide or HowTo describing changing a mod_tile/renderd-based tile server from the old style sheet to the new Carto style sheet? Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Switching tile server to CartoCSS
Ok, and where can I get the Carto style sheet that is now the basis of the OSM map? Or do I just re-fetch the Mapnik XML because it has been updated with the output from carto? Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 8/28/2013 12:28 PM, Frederik Ramm wrote: Hi, On 08/28/2013 05:55 PM, Lynn W. Deffenbaugh (Mr) wrote: I should have asked this a while ago, but is there any guide or HowTo describing changing a mod_tile/renderd-based tile server from the old style sheet to the new Carto style sheet? The Carto bit affects the writing of the style, not the using of the style; you take the style written in Carto, run it through carto (the program) once, and you get a Mapnik XML just like before. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tile server
18GB for the planet and my postgresql database is about 260GB after importing the planet into it. Add space for rendered tiles, and it's pretty demanding. If you've only got a single magnetic spindle for the 500GB, you might find it hard to keep up with the updates. I ended up migrating from a 3 spindle magnetic RAID to a pair of SSDs, one for the rendering tables and the other for the import tables. I'm looking into which tables to put on which to try to make both busy instead of just one when rendering. Lynn (D) - KJ4ERJ On 6/21/2013 12:19 AM, Kaur gill wrote: On Thu, Jun 20, 2013 at 8:42 PM, malcolm stanley a.malcolm.stan...@gmail.com wrote: out of curiosity, how much disk space are you consuming? I am setting up a tile server right now and about to ingest the planet file. I have 500 Gb of storage connected to the server: will that be enough, or will I run out of space during ingest? Any metrics on the size of a fully ingested system would be appreciated... The whole planet is at least 18GB when compressed. I think its enough. You can refer this link: http://switch2osm.org/serving-tiles/manually-building-a-tile-server-12-04/. -- Kamaljot Kaur Blog: http://kamal125130.wordpress.com/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Label font scaling
I'm running my own Tile Server (mod_tile/renderd created from packages as described at http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/ and would like to serve up a new set of tiles with larger font sizes for higher resolution displays. I read https://help.openstreetmap.org/questions/8467/font-size-and-adding-scale-to-map which describes the textsymbolizer element in osm.xml, to whit: https://help.openstreetmap.org/vote/8472/up/ 3 https://help.openstreetmap.org/vote/8472/down/ The font size comes from the style file /osm.xml/. This includes several lines on the form /|textsymbolizer name=name fontset_name=bold-fonts size=11/|/. You can edit the /size/ parameter in the wanted areas to generate a new style with other font sizes. But there's a TON of textsymbolizer's that would need to be edited. Cloudmade's tile servers support an embedded doubler in their tile URL as described at http://developers.cloudmade.com/projects/tiles/documents To get double resolution tiles use *@2x* suffix: 997@2x - this will improve map look for iPhone 4, Motorola Milestone, etc. Does anyone know how they do that? Is there a way to replace the textsymbolizer element size attributes with a variable (say Size11, Size8, SizeN) so that I can simply clone the style and edit all of the sizes in a single place? Then I could do a 1.5, 2, 4, or whatever by simply editing the sizes where the variables are declared instead of every single textsymbolizer element. Any/all help would be appreciated to accomplish label font scaling in the rendering chain with minimal manual editing work for each scale that I support. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Label font scaling
Can you save me a bunch of research and just post an example of your entity declarations (not necessarily all of them) and one or two textsymbolizer instances that use them? Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 5/31/2013 12:12 PM, Stephan Knauss wrote: I have increased the font size on thaimap.osm-tools.org http://thaimap.osm-tools.org as well. I did this by replacing the font sizes by xml entities and adjusting these. As it seems to be useful for others as well I could submit it to svn. If nobody disagrees I can submit it on Sunday. Stephan Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to schrieb: I'm running my own Tile Server (mod_tile/renderd created from packages as described at http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/ and would like to serve up a new set of tiles with larger font sizes for higher resolution displays. I read https://help.openstreetmap.org/questions/8467/font-size-and-adding-scale-to-map which describes the textsymbolizer element in osm.xml, to whit: https://help.openstreetmap.org/vote/8472/up/ 3 https://help.openstreetmap.org/vote/8472/down/ The font size comes from the style file /osm.xml/. This includes several lines on the form /|textsymbolizer name=name fontset_name=bold-fonts size=11/|/. You can edit the /size/ parameter in the wanted areas to generate a new style with other font sizes. But there's a TON of textsymbolizer's that would need to be edited. Cloudmade's tile servers support an embedded doubler in their tile URL as described at http://developers.cloudmade.com/projects/tiles/documents To get double resolution tiles use *@2x* suffix: 997@2x - this will improve map look for iPhone 4, Motorola Milestone, etc. Does anyone know how they do that? Is there a way to replace the textsymbolizer element size attributes with a variable (say Size11, Size8, SizeN) so that I can simply clone the style and edit all of the sizes in a single place? Then I could do a 1.5, 2, 4, or whatever by simply editing the sizes where the variables are declared instead of every single textsymbolizer element. Any/all help would be appreciated to accomplish label font scaling in the rendering chain with minimal manual editing work for each scale that I support. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Label font scaling
Ok, that's a good step in the right direction. But since I stated that I'm using mod_tile/renderd, how does one configure things to use that? I see that mod_tile's renderd.py invokes mapnik.render(), but without the 3rd argument which is the scale factor. Or is there a new version of mod_tile that I need to pick up? Checking further reveals that gen_tile..cpp invokes agg_renderer, but again without that optional 3rd parameter of the scale factor. I don't know which of these two modules is actually invoked by apache when tiles are requested, but since neither one does scale_factor, this looks like a mapnik feature that's not yet used by mod_tile/renderd? Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 5/31/2013 12:47 PM, Dane Springmeyer wrote: Mapnik supports an optional `scale_factor` argument for rendering that enables the 2x scale up (or any variable size) without any modifications to your stylesheet. See documentation at https://github.com/mapnik/mapnik/wiki/Scale-factor Dane On May 31, 2013, at 8:12 AM, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to mailto:ldeff...@homeside.to wrote: I'm running my own Tile Server (mod_tile/renderd created from packages as described at http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/ and would like to serve up a new set of tiles with larger font sizes for higher resolution displays. I read https://help.openstreetmap.org/questions/8467/font-size-and-adding-scale-to-map which describes the textsymbolizer element in osm.xml, to whit: https://help.openstreetmap.org/vote/8472/up/ 3 https://help.openstreetmap.org/vote/8472/down/ The font size comes from the style file /osm.xml/. This includes several lines on the form /|textsymbolizer name=name fontset_name=bold-fonts size=11/|/. You can edit the /size/ parameter in the wanted areas to generate a new style with other font sizes. But there's a TON of textsymbolizer's that would need to be edited. Cloudmade's tile servers support an embedded doubler in their tile URL as described at http://developers.cloudmade.com/projects/tiles/documents To get double resolution tiles use *@2x* suffix: 997@2x - this will improve map look for iPhone 4, Motorola Milestone, etc. Does anyone know how they do that? Is there a way to replace the textsymbolizer element size attributes with a variable (say Size11, Size8, SizeN) so that I can simply clone the style and edit all of the sizes in a single place? Then I could do a 1.5, 2, 4, or whatever by simply editing the sizes where the variables are declared instead of every single textsymbolizer element. Any/all help would be appreciated to accomplish label font scaling in the rendering chain with minimal manual editing work for each scale that I support. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 ___ dev mailing list dev@openstreetmap.org mailto:dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] PPA vs 64bit IDs
On 3/27/2013 2:03 PM, Kai Krueger wrote: Well, for Ubuntu I have created the PPA packages as linked to on switch2osm.org. Those try and take care of all the necessary steps as much as possible. To the extent even that they make many default decision on things and do other magic setup steps which violate the official packaging guidelines. However, as mentioned above, it is unfortunately difficult to create similar packages for all the different systems, as there are simply not enough developers around to do that. I'm just about to rebuild my tile server and was wondering if the PPA packages fully support the 64bit IDs? Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osmosis 32/64 ID failure
I'm running Osmosis Version 0.34 to pull and apply minutely updates to my database for use by mod_tile/renderd. These updates began failing today with the following information: Feb 9, 2013 4:01:01 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.34 SEVERE: Thread for task 1-read-replication-interval failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent 2147483648 as an integer. at org.openstreetmap.osmosis.core.util.LongAsInt.longToInt(LongAsInt.java:33) at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.init(CommonEntityData.java:142) I suspect I need to updated Osmosis which was originally installed from the instructions at http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/ which installs osmosis with the command sudo apt-get install osmosis Any hints on the command to update osmosis? I'm a *nix newbie, but very good at following cookbooks and instructions. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 11/9/2012 6:12 AM, Stefan Elspaß wrote: I tried several settings, such as ModTileRequestTimeout 500 ModTileMissingRequestTimeout 500 ModTileMaxLoadMissing 50 but still the same 404. Is there a possibility for find out why mod_tile return a 404, which timeout it runs into? Have you considered that these might be in milliseconds? And that ModTileMaxLoadMissing is the parameter that you might want to try increasing to something like 1 (10 seconds)? Or increase all of them to 1 and see if you get the effect you're after. Remember to restart renderd and/or apache when tweaking these because I'm not sure when they are actually loaded. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 11/9/2012 6:38 AM, Martin Koppenhoefer wrote: 2012/11/9 Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to: Have you considered that these might be in milliseconds? And that ModTileMaxLoadMissing is the parameter that you might want to try increasing to something like 1 (10 seconds)? Or increase all of them to 1 and see if you get the effect you're after. unless something was changed recently I think the values in mod_tile.conf are required to be in seconds. See also the default values: http://svn.openstreetmap.org/applications/utils/mod_tile/mod_tile.conf Ah well, I needed that egg on my face for breakfast anyway. It was just a guess given the similarity between his 50 value and an observed timeout of 29-49msec. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql and bbox imports
I recently switched to --flat-nodes on my osm2pgsql import and am pleasantly impressed with the improved update speed. Of course, I had to re-create the database and re-import the plane, to make the switch, but I was going to the new license and 64bit IDs at the same time, so all was well. I do the entire planet, so --flat-nodes may not be the best for your bbox server, but it might be worth reading about. Lynn (D) - KJ4ERJ On 10/12/2012 10:27 AM, Stephan Knauss wrote: Peter Wendorff writes: osm2pgsql has to store nodes outside the bbox because geometries that overlap the borders etc. should be included in the result, too. depends on the cutting algorithm used. I could live with osm2pgsql doing a hard cut as I made my bbox large enough to have some buffer. If reference completeness is a requirement it's still possible to pass it to a softcut filter before and leave away the bbox at all. Here is a description of two implemenations of a cutting algorithm https://github.com/MaZderMind/osm-history-splitter Yes, preprocessing might be faster therefore, but that might depend on your system setup and where the bottleneck of your pipeline is, as the cutting process faces the same problem here: it runs several times over the input file to find dependent nodes for ways that are partly in the extracted target area. My problem is that a database which was always around 2GB grow to 40GB during the import process and this is killing my vserver. It simply can't cope with the I/O load. I used the asia extract as my input file as I did before but now the nodes table contains data 90 degrees away from my bounding box. My setup documentation does not mention anything that I manually cleaned up the nodes table after import. So it could have been a change in osm2pgsql as well. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Planet Torrents (was: osm mirror at spline)
On 9/26/2012 4:10 PM, Paul Norman wrote: .pbf would be good - it should be used in preference to .osm.bz2 whenever possible. There are torrents set up at http://osm-torrent.torres.voyager.hr/ already that use webseed. Perhaps you could be added to those? But I believe the torrents (or their tools) need to be updated. planet-latest.osm.bz2.torrent, created 9/22 has planet-120822.osm.bz2 inside. planet-latest.osm.pbf.torrent, also created 9/22 has planet-120822.osm.pbf inside. I'd love to keep a seed running for these, but not until I can find ones that actually host the new planet files. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql -C vs --flat-nodes
BTW, I believe you can access my OSM machine's munin graphs at: http://ldeffenb.dnsalias.net:6360/munin/localdomain/localhost.localdomain/index.html If this doesn't work, please let me know and I'll see what I've not got configured correctly. Lynn (D) - KJ4ERJ On 9/19/2012 9:38 AM, Lynn W. Deffenbaugh (Mr) wrote: Greetings developers, I'm in the process of attempting to load the newly licensed planet and have recently learned about --flat-nodes in osm2pgsql. I'm trying to make use of this feature to reduce the disk space consumption of the --slim updatable database, but I'm having issues getting enough memory allocated to my VMware virtual machine to complete the import. It runs out of memory querying the pending_ways. I've looked through the code and it appears that using a -C 14000 in conjunction with --flat-nodes may be redundant as they're both attempting to speed up access to a node's coordinates, the -C by keeping it completely in RAM and --flat-nodes doing a RAM-based cache of 10,000 disk-based blocks of 1024 nodes each. Granted the -C 14000 manages to hold all 1,569,263k nodes in RAM (at 98.9% full) while the --flat-nodes will only hold 10,240k nodes in RAM, so I can expect a (significant?) slowdown, but... Can I dramatically reduce, or nearly eliminate the -C node cache and let the --flat-nodes pick up the slack for the planet import? Will this work? And will it be nearly fast enough to be reasonable? Lynn (D) - KJ4ERJ PS. I've got a 6 core VM with 24, 28, and then 32GB of RAM hosted on an 8 core i7 with only 28G of physical RAM. I know I'm paging the VM. Disk configuration is one virtual drive for the root and 3 virtual drives (each on a different physical spindle) lashed as a RAID0 array for the gis DB. I'm using the following import command: osm2pgsql --slim -d gis -C 14000 --number-processes 6 --flat-nodes /mnt/SSD/flatnodes/flatnodes.osm /mnt/raid0/planet/planet-120912.osm.pbf PPS. I started with bunzip2 -c /mnt/raid0/planet/planet-120912.osm.bz2 | osm2pgsql ... /dev/stdin, but WOW is the PBF faster, especially on the node portion with a 107k/sec node rate for the bz2 and 807k/sec for the pbf. PPPS. I know I need SSD, but that's not in the $$$ picture at the moment. After the planet import is complete, I move the rendering tables onto an SSD, but I'm not sure how to tell osm2pgsql that I'd like those tables created in the alternate dataspace (called, appropriately, SSD in postgresql). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: Diffs: Hourly + Daily REWOUND to 11-July-2012
Is this a new rewind, or a very delayed notice of the rewind that occurred way back then? Lynn (D) - KJ4ERJ On 7/23/2012 12:46 AM, Grant Slater wrote: Dev, Hourly + Daily http://planet.osm.org/redaction-period/ diffs have been rewound due to an error on the 11-July-2012. The hourly diffs after sequence 2357 file: http://planet.osm.org/redaction-period/hour-replicate/000/002/357.state.txt were incomplete. Likewise for the daily diffs after sequence 98 file: http://planet.osm.org/redaction-period/day-replicate/000/000/098.state.txt were incomplete To fix: * If you use the hourly diffs: stop osmosis, replace your state.txt with http://planet.osm.org/redaction-period/hour-replicate/000/002/357.state.txt and restart osmosis. * If you use the daily diffs: stop osmosis, replace your state.txt with http://planet.osm.org/redaction-period/day-replicate/000/000/098.state.txt and restart osmosis. Extra: If you were following the daily diffs and consumed sequence 2358 Timestamp: 2012-07-11T15\:00\:00Z between 11th July 15:00 GMT/UTC and 11th July 17:30 GMT/UTC, then you will unfortunately have some invalid data. I have an untested 'patch' diff available, contact me offlist or see below. If you need help, likely best to ask in #osm-dev on OFTC. http://irc.openstreetmap.org/ Regards Grant Part of OpenStreetMap sysadmin team. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: Diffs: Hourly + Daily REWOUND to 11-July-2012
On 7/23/2012 1:13 AM, Grant Slater wrote: New rewind... This is the first rewind of the hourly and daily diffs. Hourly: sequence: 2626 - 2357 Daily: sequence: 109 - 98 There was a rewind of the minutely diffs for a short period on the 11th July 2012. Sequence 141373 - 141272: Ah, now I understand my mistake. Missed the detail of Hourly and Daily, but anyone that did the Minutely rollback back on the 11th is ok through this one, right? Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Attribution string
On 7/14/2012 4:57 PM, Kai Krueger wrote: OK, I have had a first attempt at implementing the tilejson spec in mod_tile and have committed it to svn. The URL I chose for now is http://[hostname]/[styleURL]/tile-layer.json http://localhost/osm/tile-layer.json It provides the name, schema, description, attribution, min/maxzoom and tiles attributes. The values for description, attribution and tiles is taken from new parameters in the renderd.conf file. Please let me know if there are things wrong with the implementation or if it can be improved, or even needs reverting. Is this running on any publicly accessible server that I would take a look at it? I tried the obvious (http://tile.openstreetmap.org/tile-layer.json), but got back a Not Found. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Attribution string
On 7/15/2012 8:30 AM, Frederik Ramm wrote: Is there an advantage in having mod_tile generate the JSON response instead of simply putting a file at the root of the tile directory that contains the desired JSON response? That's actually what I was thinking originally, provided that apache/mod_tile/whatever would serve it out of the tile-style-rooted directory. Seems more straightforward, but possibly less easily maintained, than a generate-on-the-fly. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Attribution string
On 7/12/2012 1:12 AM, Kai Krueger wrote: On 07/10/2012 01:38 PM, Lynn W. Deffenbaugh (Mr) wrote: I've been wondering if it would be possible to put a fixed URL on the tile and/or API servers that application programs could fetch to retrieve the current attribution string for that particular tile server? Something like 0/0/0.txt or some-such that we could simply code into the tile-based clients to fetch the desired string instead of coding it directly in the clients? I think this sounds like a good idea. Rather than 0/0/0.txt, I would use a name like attribution.txt or license.txt in the base directory of the tile style. It allows for different attribution for different styles on the same server while still having a meaningful name. In addition I would suggest to add an attribution.html that has the attribution string as a html snippet with the correct hyperlink to the license and contributors. If people think this is useful, I could add this to mod_tile to help standardize the attribution URL. You captured my request precisely and I like your proposed names as well. And for anyone looking to balance this off against precedent, just think of robots.txt, but at the base directory of each tile style. This would go a long way to making attribution a) easier, b) under the control of the style publisher, and c) possible to be displayed by tile clients that allow user-configuration of the tile servers. Right now, my software can't really tell whose tiles are being displayed in order to provide correct attribution because the user configures the tile server, port, and URL prototype. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Attribution string
On 7/12/2012 3:10 AM, Jochen Topf wrote: On Thu, Jul 12, 2012 at 08:45:38AM +0200, Igor Brejc wrote: Date: Thu, 12 Jul 2012 08:45:38 +0200 From: Igor Brejc igor.br...@gmail.com To: Kai Krueger kakrue...@gmail.com Cc: dev@openstreetmap.org dev@openstreetmap.org Subject: Re: [OSM-dev] Attribution string (was: Licence redaction ready to begin) Why not use TileJSON? http://mapbox.com/developers/tilejson/ +1 I hadn't known about this before but just looked at it and it seems to be a well thought-out and documented standard. Agreed, but I'm still looking for a plain text attribution in addition to a URL to refer to. Their description of attribution is apparently plain text, but their example attribution contains HTML. Seems it should have an attribution text and attribution link so as to remove the ambiguity as to use. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Attribution string (was: Licence redaction ready to begin)
I've been wondering if it would be possible to put a fixed URL on the tile and/or API servers that application programs could fetch to retrieve the current attribution string for that particular tile server? Something like 0/0/0.txt or some-such that we could simply code into the tile-based clients to fetch the desired string instead of coding it directly in the clients? Since we all have to go in and change the attribution anyway, now might be a really good time to make it a fetchable string. Lynn (D) - KJ4ERJ PS. I'm proposing a plain text string that can then be integrated into a client display here. Another fixed URL per tile server could provide the reference URL string so that said applications can make the text attribution a link going there. On 7/10/2012 3:32 PM, Michael Collinson wrote: Hi Paul, We discussed this tonight at LWG https://docs.google.com/document/pub?id=1W7eh17a7cEUilpbictKYnWJfHoxIcA1W_tPEmqvmDkc item 5 We do not think temporary dual-licensing is particularly practical or necessary provided that we allow consumers reasonable time to make their attribution string changes. We are happy to re-visit this if there is any strong reason we are not seeing. Mike LWG / From: Richard Fairhurst [mailto:richard at systemeD.net http://lists.openstreetmap.org/listinfo/dev] // Subject: [OSM-dev] Licence redaction ready to begin // // Once it is complete, we will be ready to distribute data under the ODbL // and we'll advise of that with a separate announcement. The final pre- // redaction dataset available under CC-BY-SA has now been generated at // http://planet.openstreetmap.org/planet-120704.osm.bz2 . Where data has // been redacted, any attempt to access it from the API or the site's // 'browse' pages will return a response to that effect. / To transition from tiles derived from a CC BY-SA source to tiles derived from a ODbL source will require changing the attribution and regenerating all tiles so that CC BY-SA data is not mis-represented as being ODbL data. What time period will data be available under dual licenses so that tile servers will not have to reload their database then immediately delete every cached tile when changing their attribution? When I discussed this with Frederik awhile back I suggested publishing the first post-redaction planet as dual-licensed as well as one week of diffs to allow tile servers to continue to serve old CC BY-SA tiles and re-render over the course of a week. At the very least I would suggest distributing the first post-redaction planet as dual-licensed for the simple reason that all the information to create it will be available as CC BY-SA. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Table/Index Access (for SSD)
Greetings, Has anyone done any work on which tables and/or indices are most effectively targeted at SSD media for those of us that can't afford enough space to hold the entire global DB for osmpgsql/renderd/mod_tile updates and access? My current DB is just under 300GB (yes, it needs a VACUUM), and I'm trying to figure out if 120GB or 240GB worth of SSD could be employed for selected tables to improve diff application as well as rendering performance, primarily the latter. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Table/Index Access (for SSD)
On 6/18/2012 3:31 PM, Frederik Ramm wrote:
Use
select pg_size_pretty(pg_total_relation_size('tablename'));
to find out how much your tables need. If your database is nothing
fancy then planet_osm_{point,line,roads,polygon} will total about 90
GB including their indexes (which pg_total_relation_size takes into
account). These are the *only* tables used for rendering so if you
migrate these onto SSD, your rendering will be almost as fast as on a
100% SSD system, with only small improvements to the update process.
Very good. That's exactly what I was looking for. In my DB, I get:
planet_osm_point: 4369MB
planet_osm_line: 41GB
planet_osm_roads: 8530MB
planet_osm_polygon: 34GB
for a total of just under 88GB so a 120GB SSD should do the job. Now,
to see if I can figure out how to split up the DB to assign different
tables to different places
And I gather that the remaining planet_osm_* tables are the slim
support to allow for diff-based updating?
planet_osm_nodes: 97GB
planet_osm_rels: 4261MB
planet_osm_ways: 109GB
Lynn (D) - KJ4ERJ
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] render_list vs generate_tiles.py
render_list talks to the renderd and can be used to pre-populate the meta-tiles an apache/mod_tile-based tile server. generate_tile renders tiles direct as individual PNGs suitable for either direct access or static serving. I use render_list and wrote a script to generate the desired tile numbers into a render_list input file to prime my tile server, also built from the switch2osm instructions. Lynn (D) - KJ4ERJ On 6/17/2012 8:39 AM, Jason Clark wrote: What is the main difference between these two scripts to pre-render tiles? It appears as though render_list spits out the .meta files and generate_tiles.py generates .png files? I'm trying to pre-render tiles for a given area, for a server built off of the instructions on switch2osm. Thanks ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Map rendering issue
Without changing zoom, drag over and down South America. Watch it go white (gray). Then move on over to the blank face of Australia. That's the bug. Lynn (D) - KJ4ERJ On 6/13/2012 4:43 PM, Ian Dees wrote: On Wed, Jun 13, 2012 at 3:40 PM, Christophe Merlet red...@redfoxcenter.org mailto:red...@redfoxcenter.org wrote: Hi I have the same bug as you can see. http://tile.paulla.asso.fr/slippymap.html Ubuntu 12.04 LTS + Kai krueger paquets Import planet-120508.osm.pbf + minute diff Bug in the planet or in the setup ? I lost my hair :( Looks fine to me. Can you describe the problem you're having? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Map rendering issue
On 6/13/2012 4:56 PM, Ian Dees wrote: On Wed, Jun 13, 2012 at 3:46 PM, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to mailto:ldeff...@homeside.to wrote: Without changing zoom, drag over and down South America. Watch it go white (gray). Then move on over to the blank face of Australia. That's the bug. It looks like you included a bounding box for your initial import, preventing data above 76.6deg and below -35deg latitude from being imported to your database. Given the import command he posted here, that's not the case, but that's certainly the appearance. I had a similar problem when I first set up my tile server, and I'm struggling to remember just what I did to fix it. IIRC, I just scrapped the VM I was doing it in and started over with additional tweaks of the -C number as well as possibly some different psql settings, but I haven't been able to locate my notes. But the symptoms were identical, a complete loss of all details below some arbitrary line across the globe. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Map rendering issue
I had a similar problem in exactly the same part of the planet when setting up my tile server from the same instructions. I finally gave my VM 24GB of ram and put the 14000 (IIRC) memory setting on the import command and all was well. Lynn (D) - KJ4ERJ On 6/8/2012 8:10 AM, Jason Clark wrote: Hi Peter, the server is private although I could make it public if need be. In the meantime, two screenshots of zoomed out australia and close into sydney are below. I only see this issue if I import the planet, if I import australia on it's own the map is fine. I even tried using osmosis and combined north america and australia and it was still ok. The issue is just with the entire planet import. The import was done last night on the planet latest using the .pbf file. No updates done yet. _https://skydrive.live.com/?cid=8C07BA2CC05C3840id=8C07BA2CC05C3840%21125#cid=8C07BA2CC05C3840id=8C07BA2CC05C3840%21129 https://skydrive.live.com/?cid=8C07BA2CC05C3840id=8C07BA2CC05C3840%21125#cid=8C07BA2CC05C3840id=8C07BA2CC05C3840%21129_ On Fri, Jun 8, 2012 at 4:38 AM, Peter Körner osm-li...@mazdermind.de mailto:osm-li...@mazdermind.de wrote: Hi Jason, is the tile-server public? Can you share an url with us? If not, maybe a screenshot? Did you do an import in the close past or are you running on minutely updates? Peter Am 07.06.2012 19:31, schrieb Jason Clark: I posted earlier about what I thought was a problem with Australia data, but it isn't It looks like all data below -35 lat is missing... Anyone know what might cause this? My tile server is built from http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/ and had no issues during install. The import of the planet was fine, no errors. Thanks! ___ dev mailing list dev@openstreetmap.org mailto:dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org mailto:dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Archive.org Tile Serving
How do the tiles get updated between the (looks like) 2 servers split by GeoDNS today? Are the individual/independent rendering servers using mod_tile (or similar) or is there a central renderer that delivers tiles to both servers for serving to the end users? If the latter, then maybe the same approach could be used to populate and refresh Archive.org? Lynn (D) - KJ4ERJ On 4/25/2012 8:05 AM, Grant Slater wrote: On 25 April 2012 12:18, Stefan de Koninkste...@konink.de wrote: Hi, Last night Archive.org has set up a collection to facilitate tile serving from Archive.org. We can experiment on what the best structure of this would be. Anyone with interest in this subject please contact me. Sure. I think there is interest here. Maybe share here on dev@? On other matters... tile.osm.org uses GeoDNS and we are interested in getting additional servers. (USA, additional server in Western Europe, Eastern Europe and Eurasia would be good) Current setup here: http://dns.openstreetmap.org/tile.openstreetmap.org.html Regards Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Transactions per minute?
Is something actually going on with the redaction diffs now? Or is something else driving up the transaction IDs? I track the change in transaction ID vs the minutely sequence and instead of running between 100 and 500 transactions per minute, the past two hours have been 1,000 and 1,500 transactions per sequence respectively. But strangely enough, the diffs applied in less than 10 minutes which is more like the time it takes my system to digest 200 transactions per minute. I've been watching for a change in the update rate figuring that would be an indication of the redaction activity because I've been wondering if my marginally capable planet-wide map server would be able to digest the burst of updates that are expected. But if this is them, then I'm out of the woods and can rest easily. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Transactions per minute?
The changesets are having the expected number of nodes for this time of day (based on my recorded history), but the transaction IDs are going up much faster (txnMax from state.txt). For instance in the most recent 6 hours the txnMax was 8129933, 8140852, 8152102, 8212660, 8302266, and 8366981. This is a delta for each 60 minute interval as follows: 4/19 00:01 delta 10,919 4/19 01:00 delta 11,250 4/19 02:01 delta 60,558 4/19 03:01 delta 89,606 4/19 04:00 delta 64,715 That's the jump that I'm seeing, not in the node, way, or relation counts, but in the delta of the txnMax values which I believe are transaction IDs from the database? Lynn (D) - KJ4ERJ On 4/19/2012 12:25 AM, Paul Norman wrote: From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Sent: Wednesday, April 18, 2012 8:19 PM To: dev@openstreetmap.org Subject: [OSM-dev] Transactions per minute? Is something actually going on with the redaction diffs now? Or is something else driving up the transaction IDs? I track the change in transaction ID vs the minutely sequence and instead of running between 100 and 500 transactions per minute, the past two hours have been 1,000 and 1,500 transactions per sequence respectively. But strangely enough, the diffs applied in less than 10 minutes which is more like the time it takes my system to digest 200 transactions per minute. I don't believe they've started yet, and nothing looks unusual in my changeset watcher. There's been a half-dozen imports or so over the last day, but that's about normal. I've been watching for a change in the update rate figuring that would be an indication of the redaction activity because I've been wondering if my marginally capable planet-wide map server would be able to digest the burst of updates that are expected. But if this is them, then I'm out of the woods and can rest easily. Some back of the envelop calculations estimated that with the predicted schedule it would be at least 4-5 times normal traffic. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] minute dumps
See http://blog.osmfoundation.org/2012/03/27/service-schedule-march-april-2012/ Lynn (D) - KJ4ERJ On 4/3/2012 2:13 PM, Sergey Galuzo wrote: Hi, I cannot find recent minute diffs. Last one was created on April 1. Is there something wrong? Thanks, Sergey. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] mod_tile Developer list?
Greetings, Is there a specific mod_tile developer's list or forum? I'd like to propose, and implement, a few enhancements to the render_list.c tool that comes with mod_tile. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] mod_tile Developer list?
On 3/28/2012 1:44 PM, Frederik Ramm wrote: Lynn, On 03/28/2012 07:33 PM, Lynn W. Deffenbaugh (Mr) wrote: Is there a specific mod_tile developer's list or forum? No. This list is as close as it gets. Unless your suggestions break existing behaviour you're unlikely to see any resistance though! As for breaking existing behavior, one of the things is my opinion that existing behavior is broken. For anyone familiar with the tool: If you supply a list of tiles to render, they are checked against the planet loaded timestamp and only requested if the meta tile file is earlier than than planet load (dirty) UNLESS you put the -f (force) flag on the command line which circumvents the date check. However, if you use the -a option (along with -z/Z/x/X/y/Y), every tile in the specified range is requested with NO date check and the -f option is completely unused. My first proposal (and already implemented in my local copy of the source) is to incorporate the date check in the -a option which would then require a -f to be added to restore the current non-date-checking behavior, but restoring the -f (force) switch to what you (at least me) would expect the behavior to be, not-forced (date checked) if not specified, and forced rendering if specified. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: Upcoming Maintenance - Read Only
On 3/28/2012 2:31 PM, Derick Rethans wrote: On Wed, 28 Mar 2012, Phil! Gold wrote: * Grant Slateropenstreet...@firefishy.com [2012-03-28 09:51 +0100]: * planet.openstreetmap.org will be available but no new diffs will be generated until the license change is complete. Will the diffs express the license change? That is, will the diffs contain deletion or modification clauses for every object affected by the license change redaction, or will a database dump and reload from an ODbL planet file be necessary to have local ODbL database? I'd understood that you'd need a full reload; but I am hoping that we still get diffs (See my mail to the rebuild list). Or is it possible that the redaction tools could be made available to do an in-place application to our own local OSM/mapnik/mod_tile/osm2pgsql-built database? Although I rather believe that the osm2pgsql import of a planet may have dropped a bunch of the data on which the redaction tools base their decision? Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem compiling mod_tile
All you need to do is click the link: http://gis.19327.n5.nabble.com/Problem-compiling-mod-tile-tp5581228p5581228.html Lynn (D) - KJ4ERJ On 3/20/2012 5:11 PM, Stefan de Konink wrote: On 20-03-12 22:04, elbereth wrote: I followed the mod_tile tutorial in the OpenStreetMap Wiki (http://wiki.openstreetmap.org/wiki/HowTo_mod_tile). My server runs Debian 6 with g++, I got the following error, while trying to compile mod_tile. I don't see your error, but if you are running Apache 2.4 I have a patch for that :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem compiling mod_tile
I just remembered that I recently encountered a similar situation of not being able to build mod_tile following those instructions. They don't include any platform-specific configuration commands. If I recall, I had to execute ./autogen.sh followed by ./configure and then make would work for me. Otherwise, my make command was complaining that there was no makefile. This was on a Ubuntu VMware VM. Lynn (D) - KJ4ERJ On 3/20/2012 5:25 PM, yvecai wrote: I just opened a ticket 2 days ago: https://trac.openstreetmap.org/ticket/4303 Maybe you can up there, and post your error, it seems I had issue in pasting mine: it can barely be read :( I just suppress a few thing in the attached file until it works, but it may be safer to wait for a real fix by somebody understanding cpp and modtile better than me. Yves Le 20/03/2012 22:14, Lynn W. Deffenbaugh (Mr) a écrit : All you need to do is click the link: http://gis.19327.n5.nabble.com/Problem-compiling-mod-tile-tp5581228p5581228.html Lynn (D) - KJ4ERJ On 3/20/2012 5:11 PM, Stefan de Konink wrote: On 20-03-12 22:04, elbereth wrote: I followed the mod_tile tutorial in the OpenStreetMap Wiki (http://wiki.openstreetmap.org/wiki/HowTo_mod_tile). My server runs Debian 6 with g++, I got the following error, while trying to compile mod_tile. I don't see your error, but if you are running Apache 2.4 I have a patch for that :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql diff imports benchmarks
On 3/18/2012 4:07 PM, Kai Krueger wrote: sylvain letuffe wrote It turned out that the index on the pending field wasn't used at all and led to a full scan of the planet_osm_ways table, increasing noticeably the diff import. I think I have seen that one before as well. If I remember correctly, it was sufficient to run a simple analyze on the table. No need to do a re-index. It probably thinks a too large proportion of the ways are pending and therefore decides it is best to do a seq scan. Possibly because before the going over pending ways stage about 50% of ways are pending during the import. Osm2pgsql should be doing a analyze at the end of the import though, so I am not sure why this is happening. I have a planet database that has been imported and has been attempting to catch up for a while now (3 days to go) and it's been showing extreme data throughput spikes when entering the pending ways phase, so I decided to check this out. An analyze command show that a sequential scan is, in fact, being done: gis= explain select id from planet_osm_ways where pending; Seq Scan on planet_osm_ways (cost=0.00..5253263.74 rows=65592587 width=4) Filter: pending I'm not sure why, but I got an error attempting to analyze verbose planet_osm_ways as www-data, but it seems to execute as the postgres user: gis= analyze verbose planet_osm_ways; WARNING: skipping planet_osm_ways --- only table or database owner can analyze it The analyze output really didn't tell me much: gis=# analyze verbose planet_osm_ways; INFO: analyzing public.planet_osm_ways INFO: planet_osm_ways: scanned 3 of 3941416 pages, containing 983808 live rows and 589826 dead rows; 3 rows in sample, 131170601 estimated total rows But a subsequent explain certainly looks promising: gis= explain select id from planet_osm_ways where pending; Index Scan using planet_osm_ways_idx on planet_osm_ways (cost=0.00..916513.78 rows=43724 width=4) Now to wait for my next 12 hour catchup chunk to get to the pending ways phase,probably in about 6 hours or so. Lynn (D) - KJ4ERJ PS. If anyone gets this far, are those dead rows a function of my running with autovaccuum off? And should I be doing a periodic vaccuum to clean out accumulated cruft? I was thinking that not too much in the osm planet would be deleting, but maybe my assumption is incorrect? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
On 3/7/2012 4:57 AM, Peter Körner wrote: A Service that is able to provide 1. fast and scalable 2. tiled access to 3. updated data 4. around the world with a constant tile size (eg z12 or z14) 5. together with formulars to calculate the tile coordinate from lat/lon and 6. complete documentation I would expand 6 to be documentation for use as well as the ability to replicate the server environment using OSM planet data update feeds. I personally expect the restrictions on the tile servers to be extended to the API servers when enough application coders implement a way to use the API directly from thousands or millions of clients at which point they'll be instructed to fire up their own server and need more than just use-based documentation. Lynn (D) - KJ4ERJ - Trying to get my own tile server working reliably now... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] replicate-sequences tool bust?
Many of the instructions for getting an initial state.text file for mapnik updates says to use the tool at http://toolserver.org/~mazder/replicate-sequences/ I just tried it and it will only return the planet state from 2/17/2012 for any date past that time. Requests before 2/17 return an expected state, but it doesn't seem to want to go past 2/17. I tried the contact the tool owner link (maz...@toolserver.org), but my e-mail bounced. Is this the place to report this issue or is there a better place? Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Re-importing the Planet
I've got an OSM tile server based on the Ubuntu instructions that I've been running and updating with Minutely Mapnik for over a month now. I just noticed that southern Australia has issues like incomplete roads, lack of large rivers, no green spaces, and no city labels. I have no idea how or when that happened, but any hints would be welcome. But that's not really the question. What I'd like to do is re-import the latest planet definition that I'm about 4 hours away from completely downloading. Can I just use the same osm2pgsql import command that I did originally or do I need to do something special to the gis PostGreSQL database that's been updated for a month? And if I do just the original import, will the old and new maps try to fit in the database concurrently until the import is done (about doubling the DB size), or does the current data go away as soon as the new import starts? I've not seen any instructions on how to do such a thing, so answers or links would be most welcome. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql not importing routes?
On 2/27/2012 4:56 PM, yvecai wrote: Then I was confused by the: Process 0 finished processing 0 relations in 0 sec in osm2pgsql output and I did not checked in the database, but the relations were probably successfully imported before I rebuilt everything twice :(. Yeah, that line is talking about Relations that were marked Pending during the original Node/Way/Relation pass. You're looking for output lines that contain Stats: not rate. The following is output from applying updates, but the same stuff comes out of an original planet import. Node stats: total(555913), max(1650259295) in 2664s Way stats: total(47995), max(152189759) in 6128s Relation stats: total(968), max(2055178) in 13845s vs 33297 Pending ways took 3252s at a rate of 10.24/s 2476 Pending relations took 7881s at a rate of 0.31/s Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql not importing routes?
I've read that osm2pgsql doesn't really import everything from an OSM file into the DB. Specifically, http://wiki.openstreetmap.org/wiki/Osm2pgsql/schema says: Notice that relations are not imported directly. Ways which are members of relations are imported in a special manner (see description for planet_osm_line), but there is no easy way to establish a relationship between a relation and its members, or to get tags associated with a relation (unless they have ways as members). Also: Note that the mechanism of creating additional rows for each relation membership, with the tags of the relation, does *not* apply to nodes. That is, with the current scheme, there is no way to get parent relation data for a given node. which, conjunction with the following from http://wiki.openstreetmap.org/wiki/Osm2pgsql#From_source_.28generic.29, may explain the lack of the route_name column. The default style for osm2pgsql does not import all tags from an .OSM file to a pgsql database. see http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/default.style for a list of what keys are imported. I wonder if that's part of what you're running into? Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 On 2/23/2012 3:20 PM, yvecai wrote: Sorry, I took the subject from an old topic. Here an extract of my osm file: relation id=1401913 version=1 timestamp=2011-01-31T21:31:42Z changeset=7149413 uid=171657 user=yvecai member type=way ref=48769799 role=/ member type=way ref=97599611 role=/ tag k=color v=green/ tag k=name v=Haute Joux/ tag k=piste:type v=nordic/ tag k=route v=ski/ tag k=type v=route/ /relation Which contains 49 relations according to osm2pgsql output, but the following requests find nothing: select count(*) from planet_osm_line where osm_id = -1401913; select count(*) from planet_osm_line where osm_id 0; and the column route_name does not exists. osm2pgsql SVN version 0.80.0 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ANNOUNCEMENT: T@H server will go away end of February
That would make one large distributed tile server, but the issue is getting inbound connections to those T@H users running behind firewalls. They can connect OUT to get work and again to push the results out to a central server, but anyone that needs a tile would a) have to know who has it (central server index hit most likely), and b) get a connection INTO the renderer that has the tile. Lynn (D) - KJ4ERJ - Watching for a distributed tile server solution On 2/12/2012 4:09 AM, Simon Poole wrote: Possible? Sure. Good use of resources? Probably not. Simon Am 12.02.2012 10:00, schrieb Mike Dupont: Would it not be possible to have clients render tiles and then share them? mike On Mon, Feb 6, 2012 at 2:06 PM, Simon Poole si...@poole.ch mailto:si...@poole.ch wrote: Am 06.02.2012 10 tel:06.02.2012%2010:29, schrieb Matthias Meißer: Am 06.02.2012 09 tel:06.02.2012%2009:54, schrieb Sven Geggus: rambling about doing more of the same old stuff deleted If we want to do anything at all officially as a replacement, I would cast my vote clearly in favour of a data tile service with client side rendering. Simon ___ dev mailing list dev@openstreetmap.org mailto:dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- James Michael DuPont Member of Free Libre Open Source Software Kosova http://flossk.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] generate image failed
Or renderd needs to run as user www-data? Lynn (D) - KJ4ERJ On 2/3/2012 2:53 AM, Graham Jones wrote: It looks like www-data needs write permission on /var/run/renderd? Graham from my phone On 3 Feb 2012 07:37, Anwar Azulfa an...@troyans.net mailto:an...@troyans.net wrote: I have reconfigure mod_tile again, then when i try to restart apache2, renderd then execute renderd -f, image still not found this is pieces of output: Serif-Bold.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif-BoldItalic.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSansCondensed-Oblique.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono-BoldOblique.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerifCondensed-BoldItalic.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif-Italic.ttf renderd[18198]: DEBUG: Loading font: /usr/share/fonts/truetype/ttf-dejavu/DejaVuSansCondensed-BoldOblique.ttf Running in foreground mode... renderd[18198]: Starting stats thread renderd[18198]: Failed to open stats file: 2 renderd[18198]: Failed to open stats file: 2 renderd[18198]: Failed to open stats file: 2 renderd[18198]: Failed to open stats file: 2 renderd[18198]: ERROR: Failed repeatedly to write stats, giving up i have grant tiledir to www-data in order to can accessed by webserver. this is my /etc/renderd.onf *[renderd]* *;socketname=/var/run/renderd/renderd.sock* *num_threads=4* *tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET* *stats_file=/var/run/renderd/renderd.stats* * * *[mapnik]* *#plugins_dir=/usr/lib/mapnik/input* *plugins_dir=/usr/local/lib/mapnik2/input* *font_dir=/usr/share/fonts/truetype/ttf-dejavu* *font_dir_recurse=1* * * *[default]* *URI=/osm_tiles2/* *XML=/home/dewirobiatul/src/mapnik-style/osm.xml* *;HOST=localhost* what should i do to solve this ? 2012/2/3 Andre Joost andre+jo...@nurfuerspam.de mailto:andre%2bjo...@nurfuerspam.de Am 03.02.2012 04:13, schrieb Anwar Azulfa: ... -- Regards, M.Iftakhul Anwar ___ dev mailing list dev@openstreetmap.org mailto:dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Any pointers on the osm2pgsql command line differences for .pbf vs .bz2-compressed OSM data? With the call for people to quit accessing the OSM tile servers, this really does need to be better, or more currently, documented. I managed to get one running, but apparently there's better ways to do it now. To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s THAT is the kind of data I was looking for and haven't noticed. So your Ways are about 10% of the rate of the Nodes, and Relations are about twice as fast as the Ways. and there are very few Relations by comparison. There is definitely something different in my nodes going at 45K/s and my ways going at 0.12k/s. I've since aborted that run and restarted with some VM changes, but it looks like I've got a ways to go yet. Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. I'm also downloading the previous week's planet file just to see if my configuration goes faster on the Ways without the error, assuming I don't get the Node order error on the previous week's planet file. So add some swap and try again :-) I'll give that a go when the current run gets to the Ways and I see what the rate turns out to be. I'm at 705000k Nodes right now, so not too much longer. But I'm only getting 30.9k/s rate instead of my previous 45 so I think some of my change in the VM had less-than-desirable effects. Lynn (D) Simon Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr): I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page (http://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Planet pbf (was: 120111 Planet Node Order?)
Thank you both Ian and Andrew for the link. That's what I was looking for. Did I miss it somewhere in the Planet page (http://wiki.openstreetmap.org/wiki/Planet.osm) or should it be added there? Not even the PBF page (http://wiki.openstreetmap.org/wiki/PBF) seems to mention this link to the planet pbf, but only the extracts from Geofabrik.de. Lynn (D) - Making suggestions to make it easier for the next person following my path On 1/15/2012 5:28 PM, Andrew Ayre wrote: On 1/15/2012 5:28 PM, Ian Dees wrote: http://planet.openstreetmap.org/pbf/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] apmon's libapache2-mod-tile (was: 120111 Planet Node Order?)
On 1/15/2012 5:46 AM, Simon Poole wrote: Apmon has created a package that installs all the necessary bits and pieces see here http://wiki.openstreetmap.org/wiki/Ubuntu_tile_server since you've already done he install it wont help you a lot right now, but there are a couple of further tips on the page. I had to add python-software-properties in order to use add-apt-repository. I edited the wiki page to include this information as a note. When I executed sudo apt-get install libapache2-mod-tile gives Setting up openstreetmap-mapnik-stylesheet-data (0.1-r26689) ... unzip is not installed in /usr/bin/unzip, it is needed by this script I'm going to roll back to the original VM (http://www.thoughtpolice.co.uk/vmware/) and start over clean to see if it works after ensuring that unzip is installed before doing libapache2-mod-tile. Simply removing and re-installing libapache2-mod-tile doesn't seem to have re-executed setting up the stylesheet, so I have no real idea if the re-install actually did the unzip. Bottom line for apmon is that there seems to need to be a missing unzip dependency in one of the packages on which libapache2-mod-tile depends. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] 120111 Planet Node Order?
Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Actually, the instructions I was following called for -C 2048 (http://weait.com/content/build-your-own-openstreetmap-server), but when I did that I got the following error (running in a 4GB Ubuntu 10.04 VM): Out of memory for dense node cache, reduce --cache size I had to reduce to -C 1800 to make it even run. Nodes imported nice and fast (43.6k/s), but I received the following error: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency which to my novice reading implies that there was an issue in the Planet file dealing with node ordering. And the Way import is running MUCH slower than I would have thought given what I've read. 0.12k/s is WAY less than 43.6k/s. I know ways are slower, but I figured maybe 10% of the speed worst case, not this much slower. Lynn (D) PS. I haven't seen a -C 12000 recommended anywhere. And interestingly with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB available according to Ubuntu's System Monitor. On 1/14/2012 6:10 PM, Simon Poole wrote: You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/14/2012 7:56 PM, Simon Poole wrote: Believe me, if you can't allocate enough memory (you should be running a 64bit OS and have at least enough swap allocated if you don't have sufficient memory) for the cache for the import in question, it is going to be very very very very slow. With System Monitor reporting only 53% memory in use, I don't think memory is the reason for the slow Way import. The 1.329 billion nodes imported reasonably consistently and memory use hasn't changed with the transition from Nodes to Ways. I really think it has more to do with the out of order node error from the planet file, but google hasn't helped me find any reference to what that really means. Don't be confused by the Florida or whatever import running fast, that is peanuts in comparison to importing a full planet (1.4 billion nodes or so). Understood. I did the Florida import just to make sure the tool-chain worked. Didn't want to run the whole planet just to test it. Lynn (D) Simon Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to schrieb: Actually, the instructions I was following called for -C 2048 (http://weait.com/content/build-your-own-openstreetmap-server), but when I did that I got the following error (running in a 4GB Ubuntu 10.04 VM): Out of memory for dense node cache, reduce --cache size I had to reduce to -C 1800 to make it even run. Nodes imported nice and fast (43.6k/s), but I received the following error: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency which to my novice reading implies that there was an issue in the Planet file dealing with node ordering. And the Way import is running MUCH slower than I would have thought given what I've read. 0.12k/s is WAY less than 43.6k/s. I know ways are slower, but I figured maybe 10% of the speed worst case, not this much slower. Lynn (D) PS. I haven't seen a -C 12000 recommended anywhere. And interestingly with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB available according to Ubuntu's System Monitor. On 1/14/2012 6:10 PM, Simon Poole wrote: You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I go! t everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load a! nd going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) -http://aprsisce.wikidot.com/ - A user of OSM data dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non-agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 (Watching the discussion from the side-lines) On 12/16/2011 12:58 PM, Frederik Ramm wrote: Andy, On 12/16/2011 06:40 PM, SomeoneElse wrote: http://wtfe.gryph.de/harmless/way/9178258 suggests This object remains problematic even after looking at harmless edits. Yes. The script is not clever enough to find out what you did. It would have classed the non-agreer's change as harmless if it had been in between two *identical* versions of the object (i.e. if a full revert had taken place later). In your case, with the history and created_by coming into play, this was not the case and so the change was considered not harmless. I'll have to look into how I could improve this. The obvious choice would be: if someone adds something and whatever they added is not present in the current version any more, then that edit was harmless. However: What if the non-agreer adds the tag nmae=Aunt Gertrud's Home for Orphans and an agreer later fixes this to name=Aunt Gertrud's Home for Orphans ... the simple analysis sketched above would say clearly the non-agreer's change is harmless because the nmae tag is not present any more. But in this situation that would be wrong (I think). So while in your case the harmlessness is obvious to the human eye, I struggle to find a good algorithm that captures it. Any ideas? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
On 12/16/2011 4:14 PM, Paul Norman wrote: From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Subject: Re: [OSM-dev] Harmless edits My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non- agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. I cannot agree with your view that the responsibility for agreement transfers to the most recent editor. As an example, xybot fixes some common spelling errors in tags, as well as some other common mistakes. 'bots are one thing, manual edits are another. If a live editor fixes the nmae= to name= then would not responsibility transition to the entity that made the decision to implement said change? Lynn (D) - Just trying to understand ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 12/1/2011 8:39 AM, Richard Fairhurst wrote: Source material: https://github.com/kothic/kothic-js/wiki/Tiles-format https://github.com/kothic/kothic-js/wiki/How-to-prepare-map-style Just found the following at the first link: All coordinates of features should be Spherical Mercator integers. To save bandwidth and disk space, they are calculated to be relative to tile boundaries: [0,0] represents lower right corner, and [/granularity/, /granularity/] represents upper right. Regarding [0,0] is the lower right and [g,g] is the upper right, where does the left come in? Typo, or really intended to be that way? Lynn (D) - KJ4ERJ - Author of APRSISCE and looking for a C lightweight local renderer ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mod_tile - apache 404 - no tiles please help
NicoG wrote: BUT, when i ask http://localhost/osm_tiles/0/0/1.png for example i have a 404: [Thu Nov 04 16:19:39 2010] [info] [client 192.168.241.1] tile_translate: uri(/osm_tiles2/0/0/1.png) [Thu Nov 04 16:19:39 2010] [info] [client 192.168.241.1] tile_translate: baseuri(/osm_tiles2/) name(default) [Thu Nov 04 16:19:42 2010] [debug] mod_deflate.c(615): [client 192.168.241.1] Zlib: Compressed 298 to 229 : URL /osm_tiles2/0/0/1.png Why does the 'tile_translate' doesn't continues to run like in the case 0/0/0 ? Possibly because zoom level zero only has 0/0.png? If you go to zoom level 1, you might have a better chance at getting tiles 0,1 as in 0/0/0.png is valid. 1/0/0.png, 1/0/1.png, 1/1/0.png, and 1/1/1.png should all be valid. 0/0/1.png is not a valid tile URL. Lynn (D) - Author of APRSISCE, an OSM tile consumer ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] McNav vector rendering? (was: Fast exact ( mobile ) routing)
VeaaC FDIRCT wrote: Nevertheless, the current rendering plugins rely either on Mapnik or the OpenStreetMap tile server. To visualize your own data format you would either have to write your own Mapnik stylesheet or wait until a vector rendering plugin is available. Do you have plans to write a vector rendering plugin to remove the dependency on local or remote tiles? I'm working on an amateur radio application called APRSISCE that runs on Windows Mobile and Win32 and would love to be able to query for XML and do local map rendering rather than fetching OSM tiles as I do now. With a vector rendering solution, I can do arbitrary scaling instead of being restricted to the 2x per zoom level that OSM tiles provide. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM in a C apps
I've done the same thing with an amateur radio APRS client that uses OpenStreetMap.org maps. I learned about the tile names at the following URL and did the code to download the appropriate PNG tiles at the selected zoom level and stitched them together for display. Slick as pie! http://wiki.openstreetmap.org/index.php/Slippy_map_tilenames Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile Atton Jonathan wrote: Hello, I am not an openstreetmap user, consequently I do not know a lot about it. I am written a C application and I wish to integrate a map. My application is free (license LGPL), consequently I plan to use a free map and openstreetmap is the perfect choice for this. I am a developer in the project Enlightenment (windows manager + set of libraries + application) and I want to write a gui widget for the project and my application. Today, what is the best way to display OSM in a C application ? I saw Mapnick, I need to look deeper in this way but boost is required and I preferred to avoid big dependencies. What type of data should I use, download a jpeg/png directly ? or the data and create the map (this is what Mapnick does if I understand well) ? -- Regards. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Coordinate to Pixel at lowzoom
Greetings, I've searched the list archives, but didn't spot this question, so here goes. I'm developing a client that uses OpenStreetMap tiles and overlays coordinate (lat/lon) information above it. It's custom C coded, so I can't use any of the libraries out there. I understand the discussion on the following URL about the tile names and the long2tilex, lat2tiley, tilex2long, and tiley2lat functions. http://wiki.openstreetmap.org/index.php/Slippy_map_tilenames However, at zoom levels less than 4 (or so), I get a worsening offset to the north and south of the equator as I map objects onto the maps. You can see this effect at the following URL: http://tinyurl.com/OSMLowZoom or http://ldeffenb.dnsalias.net.nyud.net/OSM/LowZoom.htm I'm using the floating-point X/Y value scaled by 256x256 to position the X/Y within a single tile. This works fabulously for the higher zooms, but, as you can see, completely misses the continental land masses for the lower zooms. Is there a different/better set of equations to use at lower zooms? Or is the project just that strange? Lynn (D) - Author of APRSISCE for Windows Mobile - An Amateur Radio tool ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
