Re: [OSM-dev] Grid lines
On Mon, 17 Jan 2011, Andy Allan wrote: My approach would be to generate a table with the gridlines in it, but I haven't tried. I was starting to think that might be the way, but was just curious if anyone had come up with a better programmatic method (e.g. a fancy select sql statement that generates the appropriate geometries on-the-fly or similar, since they are regular repeating geometries - this should be reasonably easy for lat/lon grids but the other types aren't quite as regular so would require some thought) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Grid lines
I've been Googling around for pointers on sensible ways to do grid line overlays for OSM in Mapnik and haven't really come up with anything useful. I'm just wondering if anyone else has had a go at doing this, and if so what method they used. I'm mostly interested in a lat/lon grid, OS grid and UTM at the moment. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Check out my photos on Facebook
On Wed, 13 Oct 2010, Chris Browet wrote: Isn't facebook becoming the biggest "legal" botnet ever created? Its about time the webmail providers started banning people stupid enough to hand their authentication credentials out to a third party (facebook). I'd hope that doing so is breaking the webmail provider's T&Cs. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Important information for OpenPisteMap users
Some OpenPisteMap bits are changing and this may affect third party software. 1. The tiles have moved to http://tiles.openpistemap.org. They are currently being served from the old location as well, but this will be discontinued in the following phases: i) After a few weeks, tiles served from the old location will get a "please update your software" watermark applied to them. ii) A few weeks later, the old location will start returning "404 Not Found" If you are operating a web site that embeds OpenPisteMap using the map.js and opm.js scripts served directly from the OPM server then you *probably* don't need to do anything. Other software will need updating. 2. Owing to abuse by a small number of users (no, using a robot to pull down a third of a million tiles a week is not acceptable behaviour), OpenPisteMap will start banning people who download more than 20,000 tiles in a day or 40,000 in a week. As a point of comparison, the whole 3 Vallies resort at all zoom levels covers around 9,000 tiles, so there shouldn't really be much need to exceed these limits. If you have a specific reason why you need to then please discuss it with me - bandwidth isn't free and a small number of users are disproportionately using a lot of it. Also, a polite request to robot writers: it is generally considered good practice for robots to identify themselves in the UA string of their requests. This is desiarable so that the authors of buggy/misbehaving robots can be contacted and therefore fixed, and also so that website operators can properly estimate the cost associated with various clients. The vast majority of robot traffic that hits OpenPisteMap spoofs a "real browser" UA string (most commonly Opera, but most of the common browsers are represented too). If you're responsible for writing one of these robots, please consider fixing your software and presenting a suitable UA string. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Contours, hillshading and void filling
On Mon, 1 Mar 2010, Andy Allan wrote: > I'm pretty sure that gdal_translate here is using the .hdr and .prj > files - otherwise where is the location information coming from? Hmm.. Further investigation shows that it is magically using the .hgt and .prj when reading a file with a ".bil" extension (even though it is identical to the .hgt file). However, if you point it at the .hgt file instead then gdal_translate seems to parse the location data out of the file name anyway. The only difference appears to be that the resulting file contains "AREA_OR_POINT=Point" instead of "AREA_OR_POINT=Area". It seems that gdal_fillnodata.py outputs a tiff with "AREA_OR_POINT=Area" if I point it at a .hgt file too... (Still confused why it doesn't want to work on tiff inputs though) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Contours, hillshading and void filling
I'm having a go at hillshading with the SRTM data at the moment. Obviously the data needs to be void-filled before being shaded, so I looked at the gdal_fillnodata.py script that ships with gdal 1.7.1. This script doesn't appear to work when I point it at the GeoTIFF files (I end up with the voids still in the output file), but it does work if I use .hgt files for input. I assumed that since the .hgt files hadn't been processed by the srtm_generate_hdr.sh script that the resulting data would be incorrectly projected, but gdalinfo tells me that it contains the same projection data as the .tif files that srtm_generate_hdr.sh produces. So I've been looking at srtm_generate_hdr.sh to see exactly what it does, and I have to say that I'm extremely confused: it appears that it creates a .hdr and .prj file, and then does absolutely nothing with them - the .tif is produced by the following: cp $TILE.hgt $TILE.bil gdal_translate -ot Int16 $TILE.bil $TILE.tif Can anyone shed some light onto this? >From reading the srtm_generate_hdr.sh comments, it seems that even without the correction the script is supposed to be doing, we're only talking about an error of around 45 metres (i.e. the difference between the centre and edge of a pixel) so in the grand scheme of things it probably doesn't matter whether it is working or not, but it would be nice to know why I'm bothering to run it when generating contour lines. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Thu, 21 Jan 2010, Brett Henderson wrote: > It must be the JVM ... The only socket stuff Osmosis does is making HTTP > requests, it never listens on sockets. I'm very surprised it's listening on > all interfaces though. I know there's some debug and profiling type ports > that Java 6 opens but I thought they were only available locally and I > (perhaps wrongly) assumed they were secured in some way (eg. You can connect > using the jconsole tool and get stats on running JVMs). It is possible that the JVM will check the source address and drop remote connections... seems a bit silly though. For what its worth, my Java version info is: java version "1.6.0" OpenJDK Runtime Environment (build 1.6.0-b09) OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Wed, 20 Jan 2010, Brett Henderson wrote: > Matt's suggestion of running under strace might be the way to go here. I've > never seen anything like this before. Got it! For some reason, one of the threads creates a PF_INET6 socket on an ephemeral port, listenning on all network interfaces. The same thread then creates a connection to that socket. Due to a misconfiguration on my server, ip6tables was set to drop looped back IPv6 traffic, so it was sitting there waiting for the connection to time out. Fixing the ip6tables config to allow looped bak IPv6 traffic has resolved the problem. Some odd notes: 1. This connection can't be especially important since it was able to carry on working just fine without the connection present. 2. Since it is on an ephemeral port, the socket is presumably only be used for internal communication between threads - as such, using a socket that is listenning for connections on all network interfaces seems like a really bad idea. This sort of stuff should be done with unix domain sockets, not public network sockets. 3. Of course, I have no idea if Osmosis or the JVM itself is responsible for this. :) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Martijn van Oosterhout wrote: > That seems awfully long. It seems to match what other people have reported here. > You did tune postgresql as described in the > wiki before you started, right? Yep. > You did make sure you actually had > enough memory to make sure you aren't swapping in the process? 8GB of RAM. > On a > system with only 1GB of memory I could get it to complete in a day > (though that was a while back).. Were you using slim mode? I can import in under a day when not using slim mode, however you have to use slim mode if you want to be able to apply deltas and this adds a huge overhead. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Brett Henderson wrote: > The documentation may not be comprehensive and does assume some knowledge > around how Osmosis pipelines are constructed, but it is accurate and up to > date to the best of my knowledge. It does document each specific commandline parameter, but it is rather lacking in the "how to" sense - i.e. its great if you know exactly what commandline parameters you need and just need to understand how to use them, but less great if you're working out how to use Osmosis to download deltas for the first time. > "Honking great Java application" is a little excessive, we're hardly talking > about installing a JEE server here. Does this refer to the download size, > the fact that it runs on Java, or that fact that it seems hard to install. Well, I was having a bad day and was utterly failing to get Osmosis to work at all so I may have gone off on one here somewhat unnecessarilly... sorry. :) However, my personal opinion of Java is that it is generally a Bad Thing - it has certain specific uses but is massively overused in inappropriate situations. That probably coloured my judgement somewhat. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Brett Henderson wrote: > That seems odd. Are you able to point it to a debugging proxy to figure out > what's going on? Does it eventually time out, or does it hang forever? I've got it working, but it seems to hang for about 3 minutes before doing anything - doesn't contact the server in that time, it just appears to sit there. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, John Smith wrote: > Which is why you keep your system 2-5 minutes behind the DB system and > you don't have to catch up ;) Feel free to buy me a server that is capable of doing this. :) My server isn't exactly underpowered, but: 1. From a blank database, it takes several days to import the planet file in slim mode. This means that even if I start importing the planet file the moment it is available, by the time it has finished my server will have several days of deltas to catch up with. 2. At times, there is enough new data in the deltas to cause my server to fall behind. My existing system automatically switched to using hourly or daily deltas as appropriate when it has got far enough behind for them to make sense - this helps it catch up quicker. 3. Any outage is going to end up causing me to have to play "catch up" again anyway, so the faster this can be done the better. Most of this is down to I/O load, and there is only so much that can be done to boost database performance. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, John Smith wrote: > You can stuff the new diff files straight into osm2pgsql, it's all I'm > doing, I coded my own shell script to keep track of where my system is > up to and to pull new ones when they are available. I think thats what I'm going to have to do. I can't figure out how to get Osmosis to work - it just sits there doing nothing when I fire it up with "--rri --simc --wxc blah.osc". Even with "-v 9" on the commandline all I get is it sitting there saying "FINE: Reading current server state." and doing nothing more. :( I'm just exceptionally dubious about stuffing minutely deltas directly into osm2pgsql - my system struggles to keep up at times anyway, forcing it to catch up using only minutely deltas seems a recipe for disaster because my experience is that applying minutely deltas has a significant overhead when compared to daily deltas. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, Frederik Ramm wrote: > Yes, and I have explained about 1.000 times that what comes out of Osmosis You may have explained all this on this list 1,000 times (which I very much doubt), but I couldn't find any of it properly documented anywhere on the Wiki... > WILL be one single (daily or 1.73daily or 2.918daily or 0.002daily depending > on how often you call it) file. I'd wager that having Osmosis download 1440 minutely diffs and assemble them into a daily diff is going to take significantly more time than just wgetting a single daily diff. However, I can't verify my assumption at the moment because I'm having to spend my time battling (and currently failing) to get a honking great Java application (Osmosis) to work to do all this. I'm afraid this seems like a retrograde step to me, compared with the relative simplicity of stuffing diffs directly into osm2pgsql. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, Brett Henderson wrote: > While the old date based names were easy to identify, unfortunately they > don't work with replication files which are only approximately aligned with > timestamps. Initialising the replication is a little annoying due to the > disconnect between timestamp and sequence number, but as Frederik points out > from that point on the replication mechanism is simpler. With the old system, I used the daily deltas if my database was lagging a day or more behind, hourly deltas if I was more than an hour behind and minutely deltas under normal operation. As I understand it, this is no longer possible and I'll end up having to use minutely deltas all the time. It seems that this new system is going to make the already very time-intensive process of applying deltas even less efficient... Or have I misunderstood how this works? -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Sun, 17 Jan 2010, Matt Amos wrote: > for the things i'm using them for they seem to work fine with > osm2pgsql without any changes. if you're using osmosis to grab the > changes then you'll need to change the rcii/rci tasks to rrii/rri > tasks [1], though. Ok. The old hourly and minutely deltas had filenames based on their dates; the new ones don't. Is there a sensible way to figuring out which deltas you need without downloading several hundred *.state.txt files? The best I can come up with is trying to do a binary cut to reduce the number of downloads needed to find the right delta, but that's messy as hell compared to how trivial it was when we had date based names. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql deltas
I've just realised that the hourly and minutely planet deltas have been replaced by the new transaction based formats, but I can't seem to find any documentation regarding their use with osm2pgsql. Can anyone provide any info on this? -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry freaks me out
On Thu, 5 Nov 2009, sly (sylvain letuffe) wrote: > In case he is doing like me, that is : running a cron job over and over again > the same file where duplicates might acumulate. But there is a few chance he > reaches such a number, so that was probably a bad guess. Ah, ok. That makes sense. The way I handle expiry means that I get a fresh expiry list each time rather than appending to an existing one - as soon as each osm2pgsql run completes I discard any entried from the expiry list that don't have associated rendered tiles actually on the disk(*). * it's actually a bit more complicated than this since you have to handle all the zoom levels. See: https://subversion.nexusuk.org/trac/browser/openpistemap/trunk/scripts/expire_tiles.py to see how I do it. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ponderings about an improved tagging scheme
On Fri, 20 Feb 2009, marcus.wolsc...@googlemail.com wrote: >> Well, I'm not sure that the kind of objects found on a map can't be >> categorized by a sufficiently flexible system. > > Like > * objects belonging to multiple "types"? This was already covered by the proposal (and is indeed one of the driving factors behind it). > * different ways of categorizing the same information? I'm not sure what you mean. > Given the imagination of people it was proven beyond any doubt > that the collection is infinite. There are a finite number of objects in the OSM database. In any case, no tagging scheme is going to place any restriction on the number of different types of object - there is nothing stopping you deciding to use a new value for the "type" tag if no one has ever mapped that type of object before. > Strange, I was under the impression that our system was quite chaotic > and worked under the 2 rules: > 1. "just do it or be ignored" > 2. "don't break what others have build or they will get very angry" These 2 rules are at odds with eachother though. Over time, tags get depricated, new tags get created. If I'm going to get angry because you "break" the roads I've mapped by migrating them from depricated tags to new tags then all the software is eventually going to become a nightmare to maintain. You *can't* support a mess of depricated tags forever, at some point you have to draw the line and say "we need to fix the data so we can simplify the software". >> How about an orderly collection of attributes (which would cover most of >> what's going on now with things like "highway=" or "natural=") WITH the >> option of a free-form "tags" attribute for object types and attributes > that >> fall through the cracks? > > What about you start working on such an order so that it encompasses > everything > people have already done or are currently doing and we see each other again > in > say... 15 years? I'd like to point out that if everyone said "that job's so big, it's impossible", we'd never get anything done. I mean, a bunch of people creating a map of a good chunk of the world is such a big job that it is practically impossible... right? Oh wait - that's exactly what we've done, precisely because people actually got on and did it instead of telling eachother that it was such a big job that it wasn't worth attempting. Steering a project as big as OSM is like steering a supertanker - you can't just stop and change direction. But if no one ever takes a step back and looks at possible directions that it *might* be an idea to go in then the project is doomed. You can't keep taking the path of least resistance forever, at some point you have to say "what can we do to make this *better* instead of just continuing as we are", otherwise you just end up heading into a dead end. > Yes and we achieve consistency by consense. > People do like other people have already done. > Everything else has failed due to being ignored by an army of mappers. That's not true at all. Over the years the project has evolved - it is nothing like it was when it started. If everyone just stood by and said "we're not going to try and change anything because we will just be ignored" we would still be using constructs like segments, relations wouldn't exist, we wouldn't have spinoff projects like the cycle map and the piste map, etc. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ponderings about an improved tagging scheme
On Thu, 19 Feb 2009, Frederik Ramm wrote: Your concept is utterly unworkable of course with the current software landscape, Would you like to explain why it is utterly unworkable? "Instead of having a geometric object with some properties, we instead think of objects with some properties (like “this is a museum” and “this has the name Natural History Museum”) and the added property of “this object is positioned at such and such a location”. ... So the geometry is not the object itself, as it is now, but it is just one property of some kind of abstract object." I'm not sure what you mean by this - Every object on a map is a geometric object, so what are you claiming is the difference between "a geometric object with some properties" and an "object with some properties"? I believe this is indeed the way many pros are doing it - there is an object and the geometry is one of many properties of the object. It is a concept to keep in mind for the more distant future; I don't think we should aim to do it with the current implementation of relations though. I think you are somehow misunderstanding what I proposed - my idea really isn't that different from what we have now. For example, at the moment you can create some ways and group them into a relation which you tag as a bus route, or a cycle route, etc. I am essentially suggesting 2 things: 1. the tagging schemes between relations and other objects are unified. So the tags "type=road, classification=primary", etc. can be applied to either a way, or a relation consisting of ways. 2. the "type" tag can be used to define a context for normal objects, much as it does for relations. So rather than having to understand that a large number of fairly arbitrary tags, such as "highway=road" and "amenity=school" define what the object is, you now only have to know that the "type" tag is going to define what the object is. Both of these points seem compeltely workable with the current software. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Diff files - Best Approach?
On Thu, 19 Feb 2009, brendan barrett wrote: > 1. What's the difference between these two diff approaches? Not sure - I've always used the daily/hourly/minutely ones. :) > 2. Is there a best practice approach to keeping in sync? For OpenPisteMap, I write an updater script: http://subversion.nexusuk.org/projects/openpistemap/trunk/scripts/updater.py (you'll need to hack the paths in the script as appropriate). It basically tries to keep the local database in sync with the OSM server by grabbing the deltas each minute, but it also tries to be smart with whether it downloads the daily, hourly or minutely deltas based on how far behind you are. > FYI: I am not using any OSM tools to do the import, i'm writing my own > import code (i'm in a .Net environment, and I find it better for my > understanding if I do this myself). I can figure out how the files > work, I just need to know the high level stuff. Well, my updater script may not directly be that useful to you then (it uses osm2pgsql), but it might be a handy reference. I don't claim it is necessarilly a "best practice" implementation, but I have tried to be sensible with the number of requests it makes to the OSM server. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Ponderings about an improved tagging scheme
I've been thinking about ways to improve the way objects are tagged in OSM - for a long time I've seen some problems with the way we currently tag things, and I finally got around to writing down some of my thoughts on the subject. I wouldn't mind some feedback on my ideas: http://wiki.openstreetmap.org/wiki/User_talk:Steve_Hill - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [PATCH] osm2pgsql tile expiry
On Sun, 8 Feb 2009, Martijn van Oosterhout wrote: > Have you tuned the database at all according to what's on the > http://wiki.openstreetmap.org/wiki/Mapnik/PostGIS page? If you don't > increase the shared_memory setting then postgres is going to spend the > whole time swapping various indexes in and out of memory. Yes, I actually have the shared buffers set to 512MB at the moment, although Postgresql doesn't seem to actually be using anything close to that much. > Any server with 2GB of memory has to be able to keep up I think... The database server currently has about 1.2GB, so maybe it could do with some more. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [PATCH] osm2pgsql tile expiry
On Sun, 8 Feb 2009, Roeland Douma wrote: > However I am wondering is it possible to add a range of zoom levels? Since I > probably want to erase zoom 10-18. My method is to use osm2pgsql to generate a list of dirty tiles at my maximum zoom level (17) and then the script which actually does the expiry can generate a list of tiles for all the lower zoom levels from this information alone. e.g. if tile (x, y, 17) is expired, you know that you also need to expire (x/2, y/2, 16), (x/4, y/4, 15), (x/8, y/8, 14), etc. See: https://subversion.nexusuk.org/trac/browser/openpistemap/trunk/scripts/expire_tiles.py I'm still trying to improve the performance of the expire_tiles.py script, but the premise is that it looks on the disk to see what tiles actually exist and inserts a reference to them into a Postgres table together with a time stamp. This means that I can expire higher zoom levels more frequently than lower levels (e.g. we don't want zoom level 0 to be expired every time someone makes a change anywhere on the planet :) - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [PATCH] osm2pgsql tile expiry
On Sun, 8 Feb 2009, Jon Burgess wrote:
> I have just started running the diff import on the OSM tile server. The
> diff imports are a little slower than I'd ideally like but seem to be
> working.
I've got some significant database performance problems on my system in
general to be honest - I think I'm going to have to spend some time
tuning the DB. My server seems to be unable to keep up with the OSM
updates during the day (it gradually falls behind during the day and then
catches up again overnight: http://openpistemap.org/updategraph ). The
database server seems to be totally I/O bound most of the time.
One query that stands out as very slow is a waterways query generated by
Mapnik at low zooms:
LOG: duration: 19252.079 ms execute : select asbinary(way) as
geom,"landuse","natural","waterway" from (select * from planet_osm_line where
waterway IS NOT NULL or landuse='reservoir' or landuse='water' or
"natural"='lake' or "natural"='water' order by z_order) as water where way &&
setSRID('BOX3D(1066449.418634779 5997554.987368068,1105585.17711679
6036690.745850078)'::box3d,900913)
But as for the performance problems during updates, I haven't pinned down
any specific query that is causing problems, I think it's just being slow
at looking up any record in the DB.
> your
> osm2pgsql solution looks like it handles edge cases where a line crosses
> a tile with no nodes inside the tile.
That was the primary reason for integrating it into osm2pgsql - we're
already calculating all the PostGIS objects here, so it seemed the logical
place to also calculate dirty tiles.
The edge case it can't really handle very well is where Mapnik renders
something (e.g. text) some distance away from the object. At the moment,
the code expires any tiles that are 0.5 tile-widths away from the object
in order to try and reduce this problem. But when using the generated
tile list to expire tiles in lower zoom levels you haven't got as much
data available so you either have to just expire 1 tile and accept that
stuff might get cut off at the edge, or expire all the surrounding tiles
as well.
One option may be to have osm2pgsql calculate dirty tiles for *all* zoom
levels rather than just the maximum, but that's going to make the list
very big.
> I might be tempted to add a metatile flag too. Since mod_tile only cares
> about rendering 8x8 grids of tiles this can dramatically reduce the
> number of tiles that we actually need to track.
Is this not just a case of reducing the zoom level you ask osm2pgsql to
expire for? e.g. if your maximum zoom level is 17 (giving a map 131027
tiles wide), asking for zoom level 14 should give you meta-tiles instead
(the map would be 16384 meta-tiles wide since a zoom level 14 tile
contains an 8x8 array of zoom 17 tiles).
> I don't see anything wrong with you adding this into SVN. Thanks for the
> new code.
Cool. I'll make some of the suggested tweaks to the code and commit it
later today.
- Steve
xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/
Servatis a periculum, servatis a maleficum - Whisper, Evanescence
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [PATCH] osm2pgsql tile expiry
On Sun, 8 Feb 2009, Martijn van Oosterhout wrote: > I just realised. You have also created a parse_wkt, which opens up the > path to limited relation processing even in non-slim mode. (It doesn't > solve the problem completely, since the whole problem is that non-slim > doesn't store most of the info required, but for certain types of > relations it might be sufficient.) Yes, in order to mark tiles as dirty for deleted objects, I select the object from the database prior to deletion and parse it into a list of nodes. I have also cheated to some extent with relations - so that I don't need to care how osm2pgsql creates PostGIS objects for relations, I just parse the WKT which is being added to the database. This has the advantage that the object generation code can be changed without affecting the tile expiry code, although there will be a minor overhead introduced by parsing the WKT instead of integrating more directly in the relation processing code. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [PATCH] osm2pgsql tile expiry
On Sun, 8 Feb 2009, Martijn van Oosterhout wrote: > - The use of -o. Is there any other output osm2pgsql could produce > that would be more logical. Not really a big deal, the long option > expire-output is there which is good enough for me. I can't think of anything else that osm2pgsql outputs, but I'm open to suggestions for a more intuitive option. > - There a whole new file added (expire-tiles.c) which seems to have > some kind of tree structure. Do you think you could add a comment at > the beginning what kind of structure it is and any relevent invariants > (like middle-ram.c does). It is a simple binary tree of tiles' X coordinates with each node containing a binary tree of Y coordinates. I will add a comment explaining the structure. Its memory requirements are not capped, but daily diffs seem to result in a few hundred thousand dirty tiles at zoom 17, which are easilly accommodated. Running with the -e option for the initial import of the whole planet would be a Very Bad Thing though. :) I was originally intending the binary tree to be a quick hack to be replaced with a better structure (possibly a b-tree) once the rest of the code was working, but it turned out not to pose any noticable performance problems so I have left it as-is for now. > - Does middle.h need to include osmtypes.h? It makes use of struct osmNode, which is defined in osmtypes.h. I added the include so that osmtypes.h didn't need to be explicitly included before middle.h in .c files, but I can remove this again if that is preferred. > - You've got a definition of EARTH_CIRCUMFERENCE. Does that match with > what's used in the projection code? Good question. Looks like the projection code for Spherical Mercator defines an Earth radius of 6378137m. This gives us a circumference of: 2 * 6378137 * 3.14159265358979323846 = 40075016.68557848615314309804 I'm using 40075016.68, which I'd say is as close as it needs to be. Of course, this is going to break if someone is not using Spherical Mercator - maybe I should add code to exit with an error if a different projection is used (the OSM tiles are defined in terms of Spherical Mercator coordinates so the tile list generated is going to be fairly meaningless for other projections). - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] [PATCH] osm2pgsql tile expiry
xpected object type while
processing PostGIS data" << std::endl;
+delete geometry;
+return -1;
+}
+delete geometry;
+} catch (...) {
+std::cerr << std::endl << "excepton caught parsing PostGIS data" <<
std::endl;
+return -1;
+}
+return 0;
+}
+
size_t build_geometry(int osm_id, struct osmNode **xnodes, int *xcount, int
make_polygon) {
size_t wkt_size = 0;
std::auto_ptr > lines(new std::vector);
Index: osm2pgsql.c
===
--- osm2pgsql.c (revision 13558)
+++ osm2pgsql.c (working copy)
@@ -481,6 +481,8 @@
fprintf(stderr, " -W|--password\tForce password prompt.\n");
fprintf(stderr, " -H|--host\t\tDatabase server hostname or socket
location.\n");
fprintf(stderr, " -P|--port\t\tDatabase server port.\n");
+fprintf(stderr, " -e|--expire-tiles zoom\tCreate a tile expiry list for
a zoom level.\n");
+fprintf(stderr, " -o|--expire-output filename\tOutput filename for
expired tiles list.\n");
fprintf(stderr, " -h|--help\t\tHelp information.\n");
fprintf(stderr, " -v|--verbose\t\tVerbose output.\n");
fprintf(stderr, "\n");
@@ -571,6 +573,8 @@
int sanitize=0;
int pass_prompt=0;
int projection = PROJ_SPHERE_MERC;
+int expire_tiles_zoom = -1;
+const char *expire_tiles_filename = "dirty_tiles";
const char *db = "gis";
const char *username=NULL;
const char *host=NULL;
@@ -607,10 +611,12 @@
{"port", 1, 0, 'P'},
{"help", 0, 0, 'h'},
{"style",1, 0, 'S'},
+{"expire-tiles", 1, 0, 'e'},
+{"expire-output", 1, 0, 'o'},
{0, 0, 0, 0}
};
-c = getopt_long (argc, argv, "ab:cd:hlmMp:suvU:WH:P:E:C:S:",
long_options, &option_index);
+c = getopt_long (argc, argv, "ab:cd:hlmMp:suvU:WH:P:E:C:S:e:o:",
long_options, &option_index);
if (c == -1)
break;
@@ -633,6 +639,8 @@
case 'H': host=optarg; break;
case 'P': port=optarg; break;
case 'S': style=optarg; break;
+case 'e': expire_tiles_zoom=atoi(optarg); break;
+case 'o': expire_tiles_filename=optarg; break;
case 'h':
case '?':
@@ -690,6 +698,8 @@
options.mid = slim ? &mid_pgsql : &mid_ram;
options.cache = cache;
options.style = style;
+options.expire_tiles_zoom = expire_tiles_zoom;
+options.expire_tiles_filename = expire_tiles_filename;
out = &out_pgsql;
out->start(&options);
Index: middle.h
=======
--- middle.h(revision 13558)
+++ middle.h(working copy)
@@ -7,6 +7,8 @@
#ifndef MIDDLE_H
#define MIDDLE_H
+#include "osmtypes.h"
+
struct keyval;
struct member;
struct output_options;
Index: expire-tiles.c
===
--- expire-tiles.c (revision 0)
+++ expire-tiles.c (revision 0)
@@ -0,0 +1,352 @@
+/*
+ * Dirty tile list generation
+ *
+ * Steve Hill
+ */
+
+#include
+#include
+#include
+#include "expire-tiles.h"
+#include "output.h"
+#include "pgsql.h"
+#include "build_geometry.h"
+
+#define EARTH_CIRCUMFERENCE40075016.68
+#define HALF_EARTH_CIRCUMFERENCE (EARTH_CIRCUMFERENCE / 2)
+#define TILE_EXPIRY_LEEWAY 0.5 // How many tiles worth
of space to leave either side of a changed feature
+#define EXPIRE_TILES_MAX_BBOX 3 // Maximum width or
height of a bounding box (metres)
+
+struct tile_subtree {
+ struct tile_subtree * less;
+ struct tile_subtree * greater;
+ int y;
+};
+
+struct tile_tree {
+ struct tile_tree * less;
+ struct tile_tree * greater;
+ int x;
+ struct tile_subtree * subtree;
+};
+
+static int map_width;
+static double tile_width;
+static const struct output_options * Options;
+static struct tile_tree * dirty = NULL;
+static int outcount;
+
+static void add_to_subtree(struct tile_subtree ** tree, int y) {
+ while (*tree) {
+ if (y < (*tree)->y) tree = &((*tree)->less);
+ else if (y > (*tree)->y) tree = &((*tree)->greater);
+ else return;// Already in the tree
+ }
+ *tree = calloc(1, sizeof(**tree));
Re: [OSM-dev] Export dead?
On Wed, 17 Dec 2008, Tom Hughes wrote: > It doesn't work while the planet file is reloading on a Wednesday morning. Ah, ok - I hadn't realised that, thanks. - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Export dead?
The map exporter seems to be dead at the moment - it is only exporting the coastline (no roads, etc). Is anyone else having this problem? (I'm trying to export a map near Bristol - changing the zoom level doesn't seem to make any difference) - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch mode very slow.
On Fri, 10 Oct 2008, Matt Amos wrote: > looks like something is making your disk write performance suck... if > this is a hosted VM then you might be getting contention from other > users on the same host machine. i had that with bytemark, but they > were cool about it and moved my VM to a different host with less disk > activity. no problems since then. It's a VM running on my own server - there should be very little other disk activity... maybe time to do some tests and work out what's making it suck... :( Thanks. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch mode very slow.
On Fri, 10 Oct 2008, Matt Amos wrote: > does it hang, or does it just take a while? Well, strictly speaking I would probably say it takes a while, rather than actually hanging, since the CPU time used by the postgres process running the UPDATE is incrementing. But "a while" is well over a day, so it may as well be hung. :) (I haven't seen it complete the daily patch yet, and I started it going yesterday morning). > i'm running with the following package versions > postgresql 8.3.3 > postgresql-postgis 1.3.3-1 > postgresql-contrib 8.3.3 I have: compat-postgresql-libs-4-1PGDG.rhel5 postgresql-plperl-8.3.4-1PGDG.rhel5 compat-postgresql-libs-debuginfo-4-1PGDG.rhel5 postgresql-test-8.3.4-1PGDG.rhel5 postgresql-8.3.4-1PGDG.rhel5 compat-postgresql-libs-debuginfo-4-1PGDG.rhel5 postgresql-devel-8.3.4-1PGDG.rhel5 postgresql-pltcl-8.3.4-1PGDG.rhel5 postgresql-libs-8.3.4-1PGDG.rhel5 compat-postgresql-libs-4-1PGDG.rhel5 postgresql-debuginfo-8.3.4-1PGDG.rhel5 postgresql-plpython-8.3.4-1PGDG.rhel5 postgresql-contrib-8.3.4-1PGDG.rhel5 postgresql-server-8.3.4-1PGDG.rhel5 postgresql-docs-8.3.4-1PGDG.rhel5 postgis-1.3.2-1.rf > what does iostat say? (more importantly, what does `iostat 30` say if > you leave it 5 mins?) http://www.nexusuk.org/~steve/stat.txt It is running under Xen... I wonder how badly that makes the disk performance suck... - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch mode very slow.
On Mon, 29 Sep 2008, Grant Slater wrote: >> What are the postgres version requirements for patch mode? > >> = postgresql 8.2 with GIST and GIN index support compiled in. Unfortunately I'm seeing the same problem after upgrading to the postgresql-8.3.4-1PGDG.rhel5 package. The initial import of the planet goes ok with no errors, but it just hangs when importing the diffs: - [EMAIL PROTECTED] osm2pgsql]$ ./osm2pgsql -m -d gis -a -s -C 500 ~/planet/20081001-20081002.osc.gz osm2pgsql SVN version 0.55-20080813 $Rev: 9211 $ Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point Setting up table: planet_osm_line Setting up table: planet_osm_polygon Setting up table: planet_osm_roads Mid: pgsql, scale=100, cache=500MB, maxblocks=64001*8192 Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Reading in file: /home/mapnik/planet/20081001-20081002.osc.gz - The load average is 1, but nothing seems to be using much CPU so I guess it's all I/O bound. ps shows a number of postgres processes doing stuff: - 13459 ?Ss 0:00 postgres: mapnik gis [local] idle in transaction 13460 ?Ss 0:00 postgres: mapnik gis [local] COPY 13461 ?Ss 0:00 postgres: mapnik gis [local] COPY 13462 ?Ss 0:00 postgres: mapnik gis [local] COPY 13463 ?Ss 0:00 postgres: mapnik gis [local] idle in transaction 13464 ?Ds20:28 postgres: mapnik gis [local] UPDATE 13465 ?Ss 0:00 postgres: mapnik gis [local] idle in transaction 30097 ?Ss 0:00 postgres: logger process 30099 ?Ss 1:31 postgres: writer process 30100 ?Ss 0:07 postgres: wal writer process 30101 ?Ss 0:00 postgres: autovacuum launcher process 30102 ?Ss 0:06 postgres: stats collector process - - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch mode very slow.
On Fri, 26 Sep 2008, Martijn van Oosterhout wrote: > It works? I don't see how. Didn't it complain about not being able to > find GIN indexes and thus not being usable for patches? It complained that one of the postgresql-contrib packages not being installed, so I installed it. It hasn't made any complaints since, but it clearly isn't working properly. What are the postgres version requirements for patch mode? - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch mode very slow.
On Wed, 24 Sep 2008, Martijn van Oosterhout wrote: > What version of postgres? Are you running the latest version from SVN? postgresql-8.1.11-1.el5_1.1 (This is a CentOS box). The version of osm2pgsql was checked out of svn last week - revision 10564. I've just done a svn update to the head, but nothing new has been checked in to osm2pgsql. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql patch mode very slow.
I'm trying to use the new osm2pgsql patch mode - I've imported the whole of last Wednesday's planet using slim mode and I'm now trying to apply the first daily patch file. However, it is *really* slow (the patching has been running for days for just a single day's diff). I'm seeing extremely long SQL queries logged by Postgres: LOG: duration: 936198.972 ms statement: EXECUTE [PREPARE: PREPARE node_changed_mark(int4) AS UPDATE planet_osm_ways SET pending = true WHERE nodes @ ARRAY[$1] AND NOT pending; Could this be something as simple as a missing index, or is something more serious likely to be wrong? - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Holes in contours
On Tue, 16 Sep 2008, Dave Stubbs wrote: > a whole pile of the missing contour lines reappear. > Interestingly a lot of the contours are cut at tile boundaries. A very good point - I hadn't noticed that. I think completely dropping the database and starting from scratch may be in order at some point (I'll probably do that when I get around to fiddling with the new diff import stuff in osm2pgsql anyway). Too much to do, too little time. :-/ - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Holes in contours
On the piste map, I have some large holes in the contours for Canada. Initially I thought these were holes in the SRTM3 data itself, but I've noticed that the cycle map seems to have much better coverage of the area. Compare: http://www.openpistemap.org/?lat=49.6124&lon=-114.4173&zoom=14&layers=B000 http://www.opencyclemap.org/?lat=49.61379&lon=-114.42141&zoom=14&layers=B000 Is the cycle map still using the SRTM3 data for Canada, or is it using the higher resolution data sets that are available for North America? I haven't seen any similarly large holes in the contours for similar latitudes in Europe but I can't think of any sane reason why this problem would be restricted to Canada. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Installing the Rails port.
On Fri, 11 Jul 2008, Jon Burgess wrote: > There are a large number of components involved with the slippy map and > export tab. These all use data from the API, the Mapnik or Osmarender > tile servers and these all have different setup requirements. If you > want more details then you need to be more specific on what exactly you > want to achieve. Currently, OpenPisteMap uses a fairly simple web page with OpenLayers to display the Mapnik tiles. My intention is to replace that web page with one that will give me search and export functions using my existing Mapnik tiles and PostGIS database. OpenPisteMap uses the OSM dataset, imported into PostGIS from the planet file about once a month and the Mapnik tiles are rendered from that database. Presumably the bits which use the OSM API can query the main API server (no need to run my own copy of the API)? - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Installing the Rails port.
On Fri, 11 Jul 2008, Jon Burgess wrote: > If you want to run the rails code with no modification then you probably > need a database to keep rails happy but it does not need to have any > data in it. Ok, that makes sense, thanks. I just have 5 months until the ski season kicks off to get around to actually doing it now. :) - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Installing the Rails port.
On Fri, 11 Jul 2008, Andy Robinson (blackadder-lists) wrote: > If you are planning to serve up a custom map then you need to render your > own tiles, and for that you will probably want a database backend unless you > plan to deliver only a very small area of the world. I'm already rendering Mapnik tiles for the OpenPisteMap website (so I have the PostGIS database already). However, the OSM site has some nice features like the map exporter that I wouldn't mind adding so I was trying to guage what I needed to do. >From what I understand from the wiki, the main OSM data set is in MySQL, but I wasn't quite able to work out if I actually needed the MySQL DB for the Ruby on Rails website to work, or if I could do without it. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] Installing the Rails port.
I'm a bit confused by what is actually required to run the website bit of OSM (not the API - just the slippymap viewer, export tab, etc) - does it need they MySQL database (with the planet imported into it), or can it be run without? - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Google Map Maker
On Tue, 24 Jun 2008, Frederik Ramm wrote: > Now that Google have unveiled their offer, which obviously must have > been some time in the making, the words have even greater power: Us > out in the open, honestly saying what we do and what we want, and > them working for months behind closed doors on some secret project > that is then one day presented to the gasping public. This is the standard Google model I'm afraid. They are often described as an open source company, but they really aren't. Google is generally a good citizen when using existing Free projects - they generally submit code back and such good stuff. However, they are rather less open when it comes to their own projects, with most of them being closed source (e.g. Google Earth, Picassa, the search engine itself, etc.). Whilst some of their projects do have open source code, they by no means have an open development model (look at Android as an example - they've been making a big deal about it for a while, but it has essentially been created behind closed doors). I have no problem with Google working this way, but I do wish people didn't keep trying to pass them off as a Free software company. Unlike open projects such as OSM, Google try to retain as much control as they can for as long as possible - they guide the direction of the project themselves, with very little scope for the community to shape the project. > Other than that, I think it's a positive development because OSM will > automatically get some of the publicity by means of contrast. Yes, although it is a double edged sword - there is already a problem with people outside the Free software world not understanding the differences between free (as in beer) and Free (as in speech) and the real benefits the latter can bring (how often is the "why don't you just use Google's free (no cost) maps?" question heard?). This further muddies the waters for the people who didn't fully understand the freedom concept to begin with. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Volunteers needed now. OSM for Myanmar disaster relief.
On Mon, 9 Jun 2008, Christopher Schmidt wrote: > TileCache + OSM + Mapnik is probably a very poor path to take. The world > boundaries shapefiles are simply too large to reasonably expect to fit > into memory on most machines in a TileCache-like setup. Why not put all the shapefiles in PostGIS? Surely that would allow them to be processed more efficiently? (At least, I have everything in PostGIS on OpenPisteMap and it seems to work well. However, I'm not using TileCache and my Mapnik is probably somewhat out of date). - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Mapnik style updates: highway=road
On Mon, 9 Jun 2008, Thomas Wood wrote: > Since this is a new value for the highway tag, it may require also > adding to the relevant osm2pgsql places. One I can see is: > http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/output-pgsql.c#L104 Good point - I had forgotten some specific highway values were handled in osm2pgsql. I have now committed an update to osm2pgsql which adds highway=road to that table. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] Mapnik style updates: highway=road
The highway=road tag was approved last week and Osmarender has been rendering it for a few days now. I have created a patch for the Mapnik styles: http://www.nexusuk.org/~mapnik/highway-road3.patch You can see an example rendering here: http://www.nexusuk.org/~mapnik/image.png (The road which runs down the East side of the river and the North side of the motorway is tagged as highway=road). I'm not sure what the procedure is for applying patches to the Mapnik styles - if everyone is happy with it, I can commit the changes to SVN myself; or if preferred, can someone apply the above patch? Thanks. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] highway=road
On Mon, 12 May 2008, Shaun McDonald wrote: > Do you have an example rendering? There is a small area of map here: http://dev.openpistemap.org/?lat=51.68314&lon=-3.92743&zoom=15&layers=00B00 The darker grey roads, such as the ones running through the hospital are in the new style. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] highway=road
On Mon, 12 May 2008, Tom Hughes wrote: > Presumably a different shade of grey to pedestrian roads, which > are already grey? It is different, although whether it is different enough is another question. The colours can be tweaked though if people think it is necessary. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] highway=road
As discussed on the talk list, I have created a highway=road style for Mapnik (patch at the bottom of this mail) - this renders ways tagged as highway=road in a similar style to unclassified roads, but grey in colour so they can easilly be identified.. I've also created a proposal for it on the wiki: http://wiki.openstreetmap.org/index.php/Proposed_features/Generic_road Please could someone apply the below patch to the subversion tree? Thanks. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence Index: osm-template.xml === --- osm-template.xml(revision 7780) +++ osm-template.xml(working copy) @@ -1447,7 +1447,7 @@ - ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') 10 5 @@ -1458,7 +1458,7 @@ - ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') 10 5 @@ -1491,7 +1491,7 @@ - ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') 5 25000 @@ -1502,7 +1502,7 @@ - ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') 5 25000 @@ -1513,7 +1513,7 @@ - ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') 25000 5000 @@ -1524,7 +1524,7 @@ - ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') 25000 5000 @@ -1535,7 +1535,7 @@ - ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and not ([tunnel]='yes' or [tunnel]='true') 5000 1000 @@ -1546,7 +1546,7 @@ - ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') + ([highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' or [highway] = 'road') and ([tunnel]='yes' or [tunnel]='true') 5000 1000 @@ -2072,7 +2072,7 @@ - [highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'road' + [highway] = 'tertiary' or [highway] = 'residential' or [highway] = 'unclassified' or [highway] = 'road' 100 10 @@ -2082,10 +2082,58 @@ - [highway] = 'residential' or [highway] = 'minor' or [highway] = 'road' + [highway] = 'road' 10 5 +#ddd +2 +round +round + + + + + [highway] = 'road' + 5 + 15000 + +#ddd +3 +round +round + + + + + [highway] = 'road' + 25000 + 5000 + +#ddd +9.4 +round +round + + + + + + [highway] = 'road' + 5000 + 100 + +#ddd +13 +round +round + + + + [highway] = 'residential' or [highway] = 'minor' or [highway] = 'unclassified' + 10 + 5 + #fff 2 round @@ -2094,7 +2142,7 @@ - [highway] = '
Re: [OSM-dev] An Introduction
Iván Sánchez Ortega wrote: > Eeerrr... That can be done already, look at the "view"->"zoom to selection" > option in the top menu... Oops - excellent, I had looked for that but couldn't find it :) Thanks. -- - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] An Introduction
Iván Sánchez Ortega wrote: > Suggestion: have a look at JOSM plugins, in particular at the Validator > plugin. One thing that would be really useful in JOSM is to be able to "zoom" the map to a selected node. I find it quite hard to actually find the node the validator is complaining about, even though it is hilighted. -- - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] osm2pgsql(-ng)?
On Wed, 9 Apr 2008, Martijn van Oosterhout wrote: > - Direct loading of planet-diffs I've been knocking together a python script to load all the OSM data into Postgresql and generate the tables needed for Mapnik, with support for the Osmosis diffs. The problems I have at the moment are: 1. It needs a lot of disk space for the database because you need to store the entire OSM dataset rather than just the cut-down data used by Mapnik 2. The initial import of the planet data is slooowww. Partly because it is Python (which I used to make development quicker with the aim or porting to C at a later date if necessary), but mostly just due to the amount of data being stuffed into Postgres. I guess this isn't a big deal since once the initial import is done you can import diffs relatively quickly. 3. I chose to use the pgdb library to talk to Postgresql, but it turns out that it doesn't support cursors properly so this is causing me some issues when selecting large datasets from the DB. What kind of figures are we talking about for disk space and time needed to do a planet import with osm2pgsql in slim mode? I'm interested in whether you've got a better solution to me. :) - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Altitude data & (cycle) route profiles
On Sun, 6 Apr 2008, Sjors Provoost wrote: > 1 - import SRTM data for (part of) Australia > 2 - write several algorithms to estimate the height of any given > coordinate (server side), using original STRM grid > 3 - display (on the fly) an altitude profile given a route (a sequence > of nodes). Drawing done server side. > 5 - integrate with the main website (on a dev server) This sounds good to me. I'd also suggest considering implementing an API for clients to get at the SRTM altitude data. Ideas that spring to mind are: 1. A simplistic solution: clients can request the elevation at a given lat/lon and the server returns the interpolated elevation. 2. Slightly more complex: clients can supply two lat/lon tuples and the server returns a list of elevations along the line between them. 3. Even more complex: as with (2) but return the data as bezier (or similar) curves. You're going to need to be able to generate this data anyway, so making it accessible via an HTTP API should be quite trivial. > For method 1 I will write a ruby script that imports the SRTM data and > will use the source code of SRTM for inspiration. Where are you going to import it to? Consider using PostGIS since this will allow spacial referencing of the data (but you may need to experi > For long straight roads there is another problem: a road may go up and > down several times between two consecutive nodes. I will calculate the > altitude along several extra points along this way, although this may > not be very accurate. Generating parametric curves (as mentioned above) or the line between 2 points would probably be the best way of attacking this problem. > For method 2 I will calculate the intersection of each way with an > altitude contour. This method does not suffer from the > long-straight-road problem. I'm not sure why this doesn't suffer from the same problem - the only issue I can see with ways that have very sparse nodes is that long straight sections may, in fact, not be straight in the real world (which there is nothing you can do about - the results are only as accurate as the source data). > I think that method 2 will be more difficult to implement and require > much more data and processing power. That's why I moved it to the > secondary goals. If it turns out method 1 is resource intense and/or > good enough, I will skip method 2. I posted some estimated figures for storing the entire SRTM3 data set as 10m contours in PostGIS a while ago - about half a terabyte. I think your best option is to stick with the raw grid and write a system to generate parametric curves of elevation for arbitrary vectors across it on-the-fly. > Initially I would like to use Gnuplot for the visualization, because > it is simple to use. However, if time permits, I would like to > generate the profile plot client side and more fancy. The client will > simply receive a list of (x,y) coordinates from the server. ISTR seeing a PHP library for generating graphs as SVG images - can't remember what it was called but something like that may be helpful. > That separate server can use a stripped > down version of the OSM database to improve its performance. This server doesn't really need the OSM database at all - instead of sending a list of node IDs to the server, you can just send it a list of lat/lon tuples. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Altitude data & (cycle) route profiles
Sjors Provoost wrote: > There was a lot of discussion about this issue and good arguments > against my solution, which is why I included step 4: to apply > different methods and compare them. I've now included real time > calculation of altitude from SRTM data. I don't think you have explained what the benefits of your solution are? What is the benefit of including this data in the OSM database rather than having a separate "elevation server" which you can hand a list of lat/lon tuples and get a list of elevations derived from the SRTM dataset. Surveying and putting elevation data in the node is a sensible idea (but the editors should do something like erase the elevation if someone moves the node). However, what you are suggesting is not surveying for each node - you are suggesting interpolating from an existing data set and I think this will lead to inaccurate and misleading elevation data being recorded at the node. For example, the elevation data for a path along the edge of a high cliff is going to be very inaccurate because you will be interpolating between a sample on top of the cliff and a sample below the cliff. If the node were surveyed properly, you would know the actual elevation of the node instead of the inaccurate interpolated height. > If I include extra nodes with lower horizontal accuracy, there has to > be a way to tag them as such. Which means all the programs which handle the data have to handle your nodes in a special way. It doesn't seem like a good solution to me. > I am not sure whether adding extra points for long roads really has > such a big effect on the size of the planet file; if most of the > database consists of windy roads and dense cities, there may be only a > few ways that need extra nodes. I could ignore any way that has more > than 1 node every 200 meters or so. Not only are you adding new nodes (I suspect this will be a significant number since there are a lot of roads with low node frequency, such as motorways), but you are also adding several new tags to the 230 million existing nodes - this is a significant amount of data (there are about 750 million tags on nodes at the moment, so the amount you are proposing adding is a significant proportion). > I am not saying all will be fine, but I think it is worth trying and > comparing with other methods. It would be great to have a true 3D map, > especially when in a (far?) future you want to do Google Earth and KML > like things. Yes, it would be great to have a true 3D map, but I don't think you have made your case as to why you need to add this data into the main OSM database instead of keeping it separate. Also, if you want to do something like Google Earth, you're going to need elevations for all of the land, not just the nodes along ways. -- - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Altitude data & (cycle) route profiles
Sjors Provoost wrote: (Much of the below has already been said by other people, but I thought I should express my views) > 2 - estimate the altitude of each node (point) in Australia from the > neighboring SRTM points Remember to put a source tag on the data (maybe we need a source:ele tag instead to show only the elevation data came from SRTM?) I am also concerned that elevation tags will become more inaccurate as nodes are moved - you should think about adding some support into the editors to warn people when they move nodes with elevation data attached and ask them if they want to remove the elevation data. > 3 - add extra nodes along long ways with few far-between nodes I'm unconvinced this is a good idea since you're introducing nodes which could well have very inaccurate lat/lon data. Nodes are usually considered to be relatively accurate since they are (generally) positioned by using some real data (GPS track, photos, etc). What you are proposing introduces a lot of nodes that have very little relation to the real world and for which the elevation data will become inaccurate if someone moves them to better represent the real world. I'm convinced that it is preferable to keep the SRTM data separate and just calculate the intersection of the SRTM data with a way when producing an altitude profile. The SRTM data could, of course, be pulled from a server rather than having to keep the data set locally. I just don't think polluting the OSM database with many potentially inaccurate nodes is a good idea. > * apply to the whole planet If you are adding extra nodes this will massively increase the size of the OSM data set. Also, consider how you are going to handle nodes which already have an ele tag - are you going to leave them alone? What about comparing the height you calculate with the existing ele tag and do something smart if they are grossly different (e.g. tag the node with a warning that the elevation should be checked) > * update render tool to show when roads do not intersect at equal hight I'm not sure what this means - a junction is a node which is a member of multiple ways - that node is going to have a single elevation tag so all the roads using it *must* have the same elevation (there is no other way of expressing the junction). If you are talking about ways crossing without a junction (i.e. no shared node), I'm unconvinced the elevation data is high enough resolution to make a meaningful judgment as to whether the roads are at the same height as each other. -- - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] svn Account
On Wed, 2 Apr 2008, Lambertus wrote: > Here you go: http://openpistemap.org/ More specifically, there are links to many of the ski resorts here: http://wiki.openstreetmap.org/index.php/WikiProject_Piste_Maps Some of them aren't using the proposed tags though, so don't render - these are some reasonably good examples though: http://openpistemap.org/?lon=6.6681&lat=44.5731&zoom=12 http://openpistemap.org/?lon=7.1478&lat=44.1862&zoom=12 http://openpistemap.org/?lon=6.9633&lat=45.4666&zoom=12 http://openpistemap.org/?lon=6.093&lat=45.1088&zoom=12 http://openpistemap.org/?lon=7.2886&lat=46.1058&zoom=12 (Apologies to anyone trying to use OpenPisteMap earlier this morning - something went mad and used up lots of memory on the server so the render queue had got quite big) - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] SoC project idea: Wikifikation of OSM
On Mon, 31 Mar 2008, Karl Newman wrote: > Well, that sounds like a nice idea, but it's fraught with peril. You > wouldn't be able to arbitrarily select any given changeset/commit for > reversion because of edits that may have happened after that commit. There would need to be some kind of conflict resolution strategy, much like Subversion has. It would be complex, but not impossible - it would probably need a good user interface though (something integrated into JOSM?) for manual resolution of conflicts. > For example, if someone deleted a bunch of relations, then later someone > else deleted a way that was a member of one of the relations, you > couldn't simply un-delete the relations because then you'd have a > referential integrity problem. Or a more subtle example--someone deletes > a way which shares a node with another way. Then the (formerly) shared > node is moved. If you un-delete the first way, is it the same as before > the delete? From the perspective of the database way table, it is (the > node list is the same), but on the map it may be drastically different. There is no automatic solution to these problems, but they can be detected and the user informed. The user can then decide what action to take (which could be to abort the whole thing, recreate deleted/moved nodes, etc). - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] SoC project idea: Wikifikation of OSM
On Sun, 30 Mar 2008, [EMAIL PROTECTED] wrote: > I came up with loads of great ideas just to realise that Frederik Ramm has > written pretty much the same on the wiki page ?Changesets and Reverts?. I've thought for a long time that OSM needs a Subversion-style global revision number and atomic commits. So, for example, if someone commits completely broken data you can roll back that revision in its entireity. Also, being able to find out what has changed within a specified bounding box since a given date would be really useful - it would allow people to take responsibility for an area and be easilly able to check and correct changes made within that area. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
