Re: [OSM-dev] Munitely updated regional osm2pgsql database
On jeudi 3 juillet 2014, Ilya Zverev wrote:
> If, with comments above, this
> script is still not the best way to have regional minutely updates,
> please point me to an alternative.
I wasn't saying that what you did is or isn't the best way ("best" beeing
relative to a need) to have regional diffs, I was just pointing out, for you,
or for anyone with a need of "subregional diffs" who might not know, that
there exists those 2 alternatives I pointed out.
--
sly, direct contact : sylv...@letuffe.org
http://wiki.openstreetmap.org/wiki/User:Sletuffe
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
On mercredi 2 juillet 2014, Ilya Zverev wrote: > and since diffs are only available for the whole planet FYI, that isn't true anymore : http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Regionally_limited_diffs Geofabrik is providing daily diffs for "any subregion you can imagine" and french association provides minute diffs for several countries, and the list can be increased on case by cases along with a real need. -- sly, direct contact : sylv...@letuffe.org http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Notes DB dumps
On mardi 18 février 2014, JB wrote: > Hello, Hello, > If not, where can the note DB be bulk-downloaded for the planet or by > country/bbox/poly? A quick search led me to : https://github.com/iandees/planet-notes-dump Which mean than Ian work(ed|s) on such a project, I guess it isn't operational yet as a service but maybe he will explain if he needs help with it ? -- sly qui suis-je : http://sly.letuffe.org ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem with XAPI and ref based search
Le vendredi 29 novembre 2013 13:41:48, Grégory Bataille a écrit : > hello everybody, > > I have been trying to use OSM data for a while. basically, first, I'd like > to get the admin boundaries of France. Do you want to construct a geometry in the end ? if the answer is yes, we have a more specific WKT or shp geometry construction tool located at : http://polygones.openstreetmap.fr (down at the moment) supporting generic type=boundary/multipolygon construction into a MULTIPOLYGON from a relation made of ways or relation members (you have to imput a relation id for it to work) -- sly (sylvain letuffe) http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HTTPS on Nominatim
Le mercredi 13 novembre 2013 18:56:53, François Lacombe a écrit : > Hi, Hi, > I've got some issues to use it on a https webpage displaying Leaflet > GeoSearch plugin since all well known browsers are blocking http content on > https websites. (...) > Many thank for any tip. You could set up a http proxy local to your hosting. -- sly (sylvain letuffe) http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tile server
On vendredi 21 juin 2013, Lynn W. Deffenbaugh (Mr) wrote: > 18GB for the planet and my postgresql database is about 260GB after > importing the planet into it. Add space for rendered tiles, and it's > pretty demanding. > > If you've only got a single magnetic spindle for the 500GB, you might > find it hard to keep up with the updates. I ended up migrating from a 3 > spindle magnetic RAID to a pair of SSDs, one for the rendering tables > and the other for the import tables. I'm looking into which tables to > put on which to try to make both busy instead of just one when rendering. You should consider a RAID0 array at OS level, that should help improve speed while still beeing simple software side. -- sly, DWG member since 11/2012 Coordinateur du groupe [ga] http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] xapi site?
Le jeudi 30 mai 2013 23:33:29, Tac Tacelosky a écrit : > I've read and re-read the api docs at > http://wiki.openstreetmap.org/wiki/API_v0.6, and XAPI and wherever > else I could find, and hadn't seen any references to this site. I guess I'll have to improve my advertising skills ;-) The server it runs on still has power left and is meant to be used by whoever needs it. I might not have pushed too much on advertising at first since I wanted to give it time to correct bugs, but after 1.5 years, I consider it quite reliable (Well, apart from software updates requiring database re-import, but my todo list contains an automatic fail-over function I've been delaying a bit too much...) > originally I was getting all the data within the > bounding box of the segment, using the Overpass API. But that meant > the data was always cached I'm unsure about what cache you are talking about, but the 3 public Overpass API instances I know about (http://wiki.openstreetmap.org/wiki/Overpass_API#Introduction) are behaving just about the same about cache and are constantly minutely updated, so if you keep receiving the same data even after an update in the bbox I feel that even using the fr instance won't change your problem... -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] xapi site?
Le jeudi 30 mai 2013 20:27:43, Tac Tacelosky a écrit : > As our > streetview software moves closer to being demonstrable, I'd like to > not use the live API as much. You are currently quering the 0.6 API at api.openstreetmap.org ? If yes, you can easily alleviate the load until you have a working alternative with xapi or overpass API by replacing api.openstreetmap.org by api.openstreetmap.fr http://wiki.openstreetmap.org/wiki/Servers/api.openstreetmap.fr -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] announcing a new mailinglist tile-serving
On jeudi 28 mars 2013, Simone Cortesi wrote: > On Thu, Mar 28, 2013 at 5:52 PM, Kai Krueger wrote: > > Renderd can do that just fine as well. Currently there is a hard coded > limit > > of 10 in the source code of renderd, but that would be easy enough to > > change. > > can you point me at some docs about this? About rendering two different > tilesets. No one is willing to move that discussion to the new born tile-serving list ? Here follow my sample config file for renderd.conf with the first 3 styles configured in (we move the hard coded limit to 100 and are happily serving 25 different tile sets based on the same osm2pgsql schema database. The problem, if any, is more at the expiry of tiles depending on your expire strategy [renderd] stats_file=/var/run/renderd/renderd.stats socketname=/var/run/renderd/renderd.sock num_threads=8 #Merci de garder cette ligne, mais si "ça ne marche pas" pour renderd car je m'en sers ailleurs pour trouver le chemin des tuiles -- sly tile_dir=/var/lib/mod_tile/ ; DOES NOT WORK YET [mapnik] plugins_dir=/usr/lib/mapnik/input font_dir=/usr/share/fonts/truetype/ font_dir_recurse=true [noname] URI=/noname/ XML=/data/project/layers.openstreetmap.fr/mapnik-styles/noname.xml HOST=layers.openstreetmap.fr [nooneway] URI=/nooneway/ XML=/data/project/layers.openstreetmap.fr/mapnik-styles/nooneway.xml HOST=layers.openstreetmap.fr [noref] URI=/noref/ XML=/data/project/layers.openstreetmap.fr/mapnik-styles/noref.xml HOST=layers.openstreetmap.fr -- sly, DWG member since 11/2012 Coordinateur du groupe [ga] http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Call for A Proper Area Datatype for OSM
Le dimanche 10 février 2013 21:31:28, Jochen Topf a écrit : > Actually the transition is the easy part. I am uneasy with your use of the work easy ! What will be the harder part ! "Just add support in every editors, most data consumer tools and export/api side, update the api so you don't download all coastlines at once, run a script for converting while keeping history intact, still allowing reverts" is what you would call easy ? I admire your enthusiasm ;-) Maybe I would call it "the maybe easier part" ;-) Still, your blog entry is a really good introduction to problems anticipation, I just find it really optimistic, and I would bet that the "discussion part about what model to choose" would be, if not faster, at least much less risky for the contributors discovering their version of software X suddenly broke without warning... -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Planet file date unclear at planet.osm.org
Le vendredi 11 janvier 2013 15:33:10, Peter Körner a écrit : > Am 11.01.2013 15:17, schrieb sly (sylvain letuffe): > > bzcat planet-130102.osm.bz2 | head -n 2 > > Or without downloading the whole file: > > curl http://planet.openstreetmap.org/planet/planet-latest.osm.bz2 | > bzcat | head -n 2 Sure ;-) Don't worry... I wasn't going to download 25GB of data to get 20 usefull bytes ;-) I wasn't clear by using the word "trick", the shell trick itself is known to me, what wasn't, is the fact this information was there, waiting for me to just read it. Some (many ?) other osm file provider don't actually provide that information and I haven't bothered to look at every .osm file around. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Planet file date unclear at planet.osm.org
Le vendredi 11 janvier 2013 14:57:59, Toby Murray a écrit : > But the osm.bz2 file has the timestamp that you need inside of it in > the second line. No need to parse the whole file. > > bzcat planet-130102.osm.bz2 | head -n 2 > > That is indeed a very usefull trick to know. Thanks. I'll update http://wiki.openstreetmap.org/wiki/Planet -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Planet file date unclear at planet.osm.org
Hi, For those who use the planet from planet.osm.org and apply diffs afterward, pay attention to the generation date of the planet. Here : http://planet.osm.org/planet/ It says "latest" with a file generation date of 05-Jan-2013 05:08 but it isn't exactly clear when is the latest object in that file (or if there is, I'm not aware of the trick) To check the date of the latest object (beside parsing the whole file) you can go to http://planet.osm.org/planet/2013/ and check the file's name date, then take some margin *before* to take your diff import sequence number. ps: on a side note, I heard there was ongoing work to provide a file's timestamp in the pbf file itselft, is that operationnal ? or planned to be ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] production site disk use question
Le lundi 31 décembre 2012 18:25:28, Jeff Meyer a écrit : > Are there any rules of thumb about Postgres disk requirements when > importing osm data? Here is a table referencing disk size use by different osm database schema : http://wiki.openstreetmap.org/wiki/FR:Servers/Comparatif_des_formats_de_base_de_donn%C3%A9es#Comparatif osmosis schema takes 500Go of disk space for the planet file of 2012-09-15 -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to make osm2pgsql quiet?
On mercredi 19 décembre 2012, Svavar Kjarrval wrote: > Hi. Hi, > I also tried to stick '> > /dev/null' and '> /dev/null 2>&1' at the end of the osm2pgsql command > within the shellscript with no effect. This works for me, I suggest you check again for syntax errors, And try : * * * * * /path/to/osm2pgqsl -blabla > /dev/null 2> /dev/null or * * * * * /path/to/wrapper-script.sh > /dev/null 2> /dev/null -- sly, DWG member since 11/2012 Coordinateur du groupe [ga] http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Death prediction of notes on osm.org as it seams to be planned
Hi, First sorry for the FUD starting title, but I found it cool to copy journalists some times ;-) This discussion first took place on [1] and I'm sad it wasn't made more public, but that might just be that developers/admins don't want my POV after all, no problem, you can quit reading right now, that is just a Message in a Bottle sent into the ocean as I'm not going to get involve in development of notes. > Kai Krueger kakrueger at gmail.com Tue Nov 13 : > Hello everyone, > > I presume you are all familiar with OpenStreetBugs and I suspect you > also have heard that there are efforts underway to bring a similar > concept to the front page of osm.org to increase the exposure of map > bugs and notes to a wider audience of mappers and problem reporters. > The current efforts can be seen at > http://notes.apis.dev.openstreetmap.org/ I've read all of the thread + summary of talk-de pro-anonymous comments, and a general statement that OSB is quite good, let's copy it more or less and even import bugs reports from it. Let's put it strait : "Going this path will lead to the death of the notes project in medium (1y) to long term (3y)" That's my prediction. Allow me to expand my thoughts on a few things : = OSB is good = No it isn't, it "was" good. If you don't agree, then you might just not be using it at all or you are still in a niche place where it is still usefull. There were those phases : the raise, stagnation, decline, and, my prediction : frustration and death. == raise (2008->2010) == OSB filled a need and OSM mappeurs discovered it first and started to use it to add notes on a map for real errors they couldn't correct themself by lack of knowledge knowing exactly what needs to be provided to help resolve the issue. reporters and fixers were on equal numbers, reporters were following their bug reports and even fixing it themself "a form of note to self" Ratio (R) signal/noise was high, OSB was usefull == stagnation (?2011) == By advertising it more and more, by integration in JOSM, OSMOSE, mobile apps, etc. mappers thought of an easy way to get help from local knowlege anonymous bypassers. Reports started to increase in number and the frustration of useless bug reports was acceptable as R kept high. == decline (2012+) == The number of reports kept on raising even more (coming from many different new bad sources: see down there[2] ), and alas, R dropped significantly IMHO, I stopped using OSB early this year because of that. I never was a reporter but a fixer. I've allways been in addition a reporter/fixer on fixme=* tagged objects but now, I'm only using that and QA tools. OSB dust starts to accumulate in my region, bugs aren't fixed, nor closed, they linger here, leading to reporters frustration. Its only future is death. (well, in my region it is agonising as the last 2 OSB fixers have quit) = My analysis = If you are still here reading me, maybe you have seen that somewhere else, let's go for my analysis : fixme tags in the db is working for me, osb isn't anymore, why ? - visibility, technical advantages, categories ? about the same, I don't thing it has to do with that - complexity ? yes. Sad to say it this way, but it has to be complex for reporters to be usefull. Why ? because the bottleneck isn't lazy reporters writing in 4s : "please fix" The bottleneck is us, the fixers ! We need to spare that ressource. =My proposed solutions= * [2] Never ever allow mass mechanical bugs reporting. The OSB API for uploading bugs was it's last terrible error accelerating it's downfall http://lists.openstreetmap.org/pipermail/tagging/2012-November/012169.html (That is also true for fixme=* tags, I'm banging my head in the wall every time someone thinks he is smart by adding a fixme=* tag all over the planet : http://www.openstreetmap.org/browse/node/308819536/history : 200k tags) * Make it complex to report bugs, no matter how Sure, you could ask the reporter to find and write the 100th Pi decimals and that will lower the number of report, but I'm sure there are better things to ask him, some have been listed on the [1] thread. What we should try to avoid is lazy reporters - mandatory email - 5 to 8 mandatory fields --are you ? : providing a name, shop closed, bad shape, routing error --your source of information is : (free mandatory field of ~10chars min) --zoom level <16 reporting not allowed * Allow and encourage bug closing let fixers be bold and close "not clear" reports. Better a false close, than a bug lingering in there. Automatic closing could also be though of (+90d = invalid = auto-close ) ps: obviouly far too long, but when reading answers on [1] I guess my opinion if far from the gaggle and I though I needed to explain [1] http://lists.openstreetmap.or
Re: [OSM-dev] osm2pgsql multipolygon with several outer ways -> postgis MULTI?
Le mercredi 21 novembre 2012 12:26:09, Per Eric Rosén a écrit : > Hi! > > Can osm2pgsql import multipolygons with several outer ways to a single > postgis multipolygon? (...) > Is this the way it's supposed to be? > I'm fine if that's the case, just good to know. Are you using the osm2pgsql -G switch ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Status of the Mapnik stylesheets
Le mercredi 14 novembre 2012 17:34:17, Eugene Alvin Villar a écrit :
> Actually we do, and that's the data layer. Unfortunately, I think there's a
> perception that the data layer doesn't provide the gratification that
> mappers want. Or maybe we just need to highlight this layer more.
IMHO Your forgot an important feature mappers want :
"Have I done a mistake or not ?"
Even if we keep on telling them : "don't map for the renderer" (in the new
sense it has become) they still want to know if what they've done is "correct"
("correct" in a non so obvious sense where answers "is it in the wiki" "It's
always correct because you can tag how you want" are not what they expect)
+the data layers isn't available as a JOSM slippy map selector (and the plugin
"download as you pan and zoom" isn't as functional because it can hardly work
on low zooms)
--
sly (sylvain letuffe)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Status of the Mapnik stylesheets
Le mardi 13 novembre 2012 23:25:44, Paweł Paprota a écrit : > On 11/13/2012 11:13 PM, Derick Rethans wrote: > > I would rather see as much useful things rendered that make sense for > > *mappers*. Pretty tiles should also be made, but as far as I know, the > > default style that is on openstreetmap.org is for *us* - the people who > > add data. > > Well, that's the usual question about what is osm.org supposed to be. I share Derick's view, but maybe what we need is someone to "just do it" and split the problem in two maps. http://wiki.openstreetmap.org/wiki/Contributors_functionalities_wishlist#Backgound_map_with_the_most_possible_objects -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
>I would like the server to answer in these 3 seconds, but with >the tiles and HTTP 200 and not with an 404. > >I tried several settings, such as > >ModTileRequestTimeout 500 >ModTileMissingRequestTimeout 500 >ModTileMaxLoadMissing 50 > >but still the same 404. Is there a possibility for find out why >mod_tile return a 404, which timeout it runs into? I'm using mod_tile from not so long ago and all those values are taken into account fine. To troubleshoot, check your are using the "notice" log severity and logs go to error.log (but I'm unsure if you'll get what you want) next thing might code hacking/debuging... ps:just in case : did your stop and then start apache ? -- sly -- sly -- sly___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] making osm2pgsql import relations?
Le mercredi 07 novembre 2012 22:20:31, Ákos Maróy a écrit : > it hasn't on purpose. the relations I use are not used to stitch > together polygons. they are relations in the traditional sense: they > collect various data about the same thing You mean categories ? ;-) > , in this case, an airport. the > related data is a range of lines (runways), points (navigation aids) and > other stuff, like the 'airport reference point', etc. Well, then, osm2pgsql might not be your best option. You might end having other problems than just "get the relation in _rels (like connecting the relation to it's members SQL in queries) > see above, the geometries are imported fine, Yes, but not the one "linked" to the relation, but to it's ways members (check the osm_id) > I do have the points and > the lines imported that are related to these airports. what I don't > have, but would want to have, is the relation itself in PostGIS I ran a test and I can confirm as well that without "type" tag, the relation isn't imported into _rels However, whatever the value is like type=toto is enough so that the relation makes it's way to the _rels table, but not to the _polygon table. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] making osm2pgsql import relations?
Le mercredi 07 novembre 2012 21:59:26, Ákos Maróy a écrit : > sure, here is the style file: > > http://code.google.com/p/openaviationmap/source/browse/trunk/rendering/oam. > style looks good > the OSM XML file can be downloaded from here: > http://files.openaviationmap.org/dump.osm osm2pgsql only uses it's multipolygon building ability when relations have a tag "type" with value "multipolygon" or "boundary" (unless patched) relation n°25 for exemple hasn't. However, I'm unsure if it should or shound't end in the _rels table anyway. But in the end, you won't have it in _polygon wich mean you probably won't get what you want : the geometry > this is the command line I use for the import: > > osm2pgsql -d oam --bbox 16,45,23,49 -s -C 2048 --hstore-all -K -v -G -v > -m -S oam.style dump.osm Looks good (beside one extra -v) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] making osm2pgsql import relations?
Le mercredi 07 novembre 2012 21:34:47, Ákos Maróy a écrit : > Is there a simple way to have osm2pgsql import relations described in an > OSM XML file? Yes : To do nothing special. Which mean, there is a problem somewhere. Can you provide your osm2pgsql command line, style file and a simple xml test case to show the problem ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Minutely changeset feed
Hi, Le mercredi 07 novembre 2012 17:15:10, Matt Amos a écrit : (...) Is there any plan to provide something like "augmented changeset diff" ? (ie : diffs with the content of the changeset in osmchange format when closed in addition to the changeset itself ) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Hello World
Le jeudi 18 octobre 2012 22:41:26, Alex Barth a écrit : > On Oct 12, 2012, at 1:14 PM, Michal Migurski wrote: > > * Create a new map style intended to be the default face of OSM, but > > leave the current OSM.orgMapnik style as-is. It works beautifully as an > > editor's basemap due to the dense inclusion of all data. Keep it, but > > add a new one that's for non-editors to look at. > > I wonder how good the current Mapnik style is for editors. By "mapnik" I asume you are refering to the "standard" map at osm.org IMHO : it is not good enough for editing purpose. see http://wiki.openstreetmap.org/wiki/Contributors_functionalities_wishlist#Backgound_map_with_the_most_possible_objects > the problem is that the Mapnik style completely ignores `surface=unpaved` > making a significant dirt road really not look how it should on the map. Arguably, the standard map might or might not show that information if we see it as a "general purpose" map. But as an "editing purpose" map like, it would be a great addition. But several other features also would : place=isolated_dwelling sac_scale=* or trail_visibility=* comes to by mind, but it could be extended to any approved feature in the wiki or at least any approved with a certain threshold of uses. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist (community wishlist built experimentation)
On jeudi 18 octobre 2012, Kai Krueger wrote: > I would suggest putting it on help.openstreetmap.org rather than on the > wiki. For sure, a wiki for voting purpose is to be classified as one of the worst tool to do it, but it was easy to copy the first wishes from the same syntax. Using help even if better suited as a tool, would be a terrible abuse of the help.openstreetmap.org instance. And in the end, we need a bit of freedom to modify things all around without pain about users rights. I'll switch if that goes out of control -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist (community wishlist built experimentation)
On jeudi 18 octobre 2012, Paweł Paprota wrote: > It's great that you are driving this. One potential use of such list > would be defining the Top Ten Tasks list. A list with that name allready exists, therefore I don't want to say that this list "would define" the TTT However, any admin is of course welcome to feed a part of the TTT with idea from that new list. Hi Pawel, > * From my experience form of presentation matters when trying to explain > complex things to people. Right now there is a lenghty first paragraph, > there is no Table of Contents. I do admit the first paragraph is huge, but it serves the purpose of scaring away un-serious people. And I find the wiki Table of content format problematic in such case, see : http://wiki.openstreetmap.org/wiki/API_v0.7 You first read "At this point, we're brainstorming new/changing features for API version 0.7." and then you jump to real ideas, forgeting to read the "Brainstorming" importante part. But why not trying after all, that shouldn't be a big deal. > Of course there is always the question if we really care about people > who have so short attention span that they can't get through a large > chunk of text but that's another discussion... That the people I want to contribute ;-) > * Similar to above point - page title. I would suggest something that > "rolls off the tongue" Does a page named TTT has better audience than one named CFW ? I don't know, and don't really care as long as the title express what it is, but if someone finds a title fullfeeling both objectiv, I'll be glad to change ! > Perhaps something like Community wishlist or > similar? I've been thinking about that, and, well, why not. The 1st sentence should clear the fact it is not about "distributing T-Shirts to mappers" > Ironically, none other than the Top Ten Tasks list has a good example of > this, see template: > > http://wiki.openstreetmap.org/wiki/Template:TopTenTask Complex..., I don't want to attract wiki experts but contributors. > I would think about making similar template but more community-focused > so no technology list but something more useful to non-developers. Unless if you come with something nice ? (I'm a wiki noob and hate spending time to understand complex templates items) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist (community wishlist built experimentation)
Le mardi 16 octobre 2012 14:29:08, Paweł Paprota a écrit : > I think this would be A LOT of work and probably not for only one person > but I for one would love to see end-users more involved during > development phase - testing, feedback, incremental improvements and all > that. > > So the question for me is - who would be interested in doing such > work... I do, but as you said, alone isn't gone be an easy task, that's why I'll soon be proposing this on talk, with the hope to get the help of everyone. I have just no ideas if something usefull could get out of this, if that won't just turn to chaos, but I guess I'll have to try to find out. As a starting point, I have cleaned-up http://wiki.openstreetmap.org/wiki/API_v0.7 into what I consider ideas for development ranging from good to improvable with a good idea behind. I've also turned it into less technical and, hopefully, understandable to long standing non-dev mappers. However, it is not limited anymore to what people think should go into the next API 0.7 version (I think it's even worse when you ask non-techies people to find a solution themself) but to something gathering ideas of wishes they find usefull for their work as mappers, what they miss more while editing (may it be external tools, websites, software or, of course API improvements). here it is, self-explained : http://wiki.openstreetmap.org/wiki/Contributors_functionalities_wishlist If anyone here wants to give feedbacks before it goes to talk, please do. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
Hi, > That said, let's talk less about talking and your personal suspicions and > more about actual substance; Is that "substance" : http://wiki.openstreetmap.org/wiki/API_v0.7 ? > aka un-derail this thread. Got it, sorry. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
> the page says: "These are the Top Ten Tasks that the OSM System > Administrators would really like your development help on." so i think > it's unfair to say it's a list by the admins, for the admins. I admit I was unfair. I shortcuted it too much. But still, that's a list by the admins. But I haven't problems with that. What I'm asking tom is : do you wish to code at admins or community requests ? I did heard his "flameware shields are not operational enough for a mail on talk" but it might be interpreted as he doesn't want to spent too much time reading people's requests about free ponies or chinese tags, but it could also be interpreted as he wants OSM admins to assign MapBox team tasks. Or in-between : "Anyone of the osm community who knows a bit what he is talking about and be able to express reasonable requests about what he needs in his day to day mapping work" > exactly - there are no restrictions on what you should work on. the data > is open, the software is open, and you can work on whatever is > interesting to you. I know that. The question is more : "On what should the MapBox team work on, and who should decide what is this what" -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
Le lundi 15 octobre 2012 18:13:34, Paweł Paprota a écrit : > - people who do the actual programming work choose what they want to > work on. I think this is a little different here (this thread) than usually is (even with what usually happen with OSM core development) Tom started this thread with : > So, along with the big 'kicking off' blog post on MapBox[1] (...) >What are the tasks which everyone agrees on, but nobody has had the time to >tackle? > [1]: http://mapbox.com/blog/kicking-off-knight-work/ By "everyone" I assume he meant "everyone on the dev list" (or everyone of the osm community) which is another way no to say "What task do I want to work on" And reading the linked blog about the "big 'kicking off'" what I understand is that he is not saying the MapBox team is going to work on what they want but "Build in the open : to allow anybody in the OpenStreetMap community to engage and to facilitate a transparent process" In the end, what I understand of his "OSM Wishlist" thread, is that he wants to start work at "some intentionally broad, but it aims to: 1. Improve editing of OpenStreetMap data 2. Make the OpenStreetMap experience more social 3. Make it easier to get data out of OpenStreetMap" And since this is really "broad", he is willing to gather tasks, agreed by the OSM community. > they can work on what > they like, instead of working on what their boss or a customer order > them to work on. Maybe we are in between on this thread ;-) > If you propose to change it by creating a community-driven > (instead of "admin"-driven as you put it) wishlist, by any means - do > it. The operative word being "do". I'll be glad to do so, and will start it. Unless the MapBox team (and tom) doesn't want to look at such a process (in the end gathering wish never read is just going to spend my time) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
On lundi 15 octobre 2012, Tom MacWright wrote: > Hey Sly, Hi tom, > Yes and no. Everything is a trade between being totally open (announce it > on IRC and see what the crowd thinks!) and being totally productive (hole > up and just do it until you're done and then realize it's duplicated > effort). That's why I'm proposing to build a short task list of wishlist first (between those who don't want to offer free ponies [1]) and then propose it to wider audience. > For the 'we have tasks' stage, I'd like to handle this in GitHub issues, My opinion is that the wiki would be the good place for the "we decide wich tasks", and after that, okay, you'r the one doing the job, so your choice. > There are existing wiki pages for > improvements (top ten tasks, api 0.7, improving openstreetmap) which have > shown at best mixed success of staying updated and being good for the > 'actual doing things' collaboration phase. Is the wiki tool in cause ? or the rules for building those pages ? "top ten tasks" is "These are the Top Ten Tasks that the OSM System Administrators" What about the community ? This only is a todo list by the admins, for the admins coded by the admins. So far so good, but that's not a wishlist, or, at least, not a wishlist of the community. [1] api 0.7 page is a brainstorming page open to every one (no moderators, no short list, no voting sytem) and, ultimetly, no one to ever do what's writen here because this is not building tasks lists, it's gathering ideas. "improving openstreetmap" is a page I've heard for the first time, so I guess it is not geared toward the community "whishlist" If the thread you've launched here named "OSM Wishlist" implies that "OSM" is it's community, then my suggestion whould be to create a wiki page explaining it's goal, asking on the dev list to build a list of say "15 to 20 non-utopic/troll/ponies tasks" then ask the community to choose 5 out of them. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
On lundi 15 octobre 2012, Pedro Larroy wrote: > I don't see why > editing and refining a page is bad. Because in this case, this page is the documentation of how to do things in the OSM db, and if by "refining" you mean changing the way to do things, then people will map according to what they've read at one point in time leading to different ways of tagging. That's why I think that for those pages, we should first find consensus and discussion before, and, if the current state is worst than changing the page, then we change the page. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
> I guess it is pretty expensive to do topology checks > in the main OSM database triggered automatically every time when relations > are saved. I think this is the best way to do it, don't allow crap in the db so that you don't have to correct it later. But it might be unreasonable for performance reasons in which case, I don't know what's the 2nd best. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
On vendredi 12 octobre 2012, Tom MacWright wrote: > Hey all (or, well, those subscribed to dev@ - my flamewar shields are at > 50% so I'm not risking an email to talk), This only is my opinion, but wishlists shouldn't be reduced to those subscribed to dev@ Maybe start on dev, but I think a wiki page should better be suited for a summary of the ~3/5 tasks to focus on. > What do you want to see happen with OSM's software this year? So many things on my side that I don't see how we can decide this on a mailing list in one thread. Better list, then short list, then vote (or kind of) and focus on few tasks. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
> And something else: Somebody recently started a collection of mp testcases > and put it in a github repository. I forgot where that was but maybe > somebody > can dig that up and join in that work. That would be interesting to include this to my wiki page to display real example cases in the .osm format for developers. Any clues where we can find those mp testcases ? -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
hi, > Perhaps a useless idea from my side. I would also like to hire you for an > hour or two for quick reading some other OGC specifications for me. I'm far from an expert for that ;-) Perhaps you mean that the OSM wiki page I've created should be made more explicit for developers (or another page more technically oriented) about this OGC simple feature standard ? > I tried ST_MakeValid and it seems to be able to convert also the two > touching inner ring case into a valid simple feature (...) > "GEOMETRYCOLLECTION(POLYGON((-139 420,71 418,59 273,-156 272,-139 > 420),(-46 312,-5 314,-4 371,-46 370,-89 370,-92 313,-46 > 312)),LINESTRING(-46 312,-46 370))" The fact that you end with a GEOMETRYCOLLECTION (with a LINESTRING) and not a MULTIPOLYGON after ST_MakeValid express the fact that the touching inner case isn't valid for OGC. But it has always been accepted as the OSM exception to the standard, because it would be more complex for mappers to do it in compliance with OGC standard. > Perhaps examples B and F could also contain at least two ways? If there is > only one way, why to make a multipolygon relation at all? You are right, both B and F (and 3 and 5) could and should be tagged as a simple closed way without relation at all. But my "polygon(s) validity" page could also be extended to non-relation polygons as well. 3 and 5, even if tagged without relation should (that's my opinion) still be considered invalid geometries. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
> By the way, perhaps there could be WKT presentations about what would be
> the simple feature solutions of the examples on page
> http://wiki.openstreetmap.org/wiki/Relation:multipolygon/validity.
Only if OSM multipolygons are still intended to be considered valid if one
valid OGC WKT representing them exists, else it doesn't matter anymore to know
the WKT presentation.
Also I probably wont take the time to do it (unless really wanted ?) since WKT
is really technically oriented and not for the usual contributorx. And for
developers, in such simple examples, a quick read of the OGC specification
should give them easily a valid WKT prestentation if it exists.
> Example
> http://wiki.openstreetmap.org/wiki/File:Multipolygon_Illustration_8_shape_w
> ith_intesection_point.svg is hard to convert into a valid simple feature if
> it is drawn as one way ABCDEBF.
It really depends how you are defining "hard" ;-)
Maybe if you intend to code the algorithm from scratch in C yes, but if you
have access to postgis 2.0 it is as simple as :
SELECT ST_AsText(ST_MakeValid('
POLYGON((-1 -1, -1 0, 1 0, 1 1, 0 1, 0 -1, -1 -1))'
));
http://blog.opengeo.org/2012/03/23/postgis-2-0-new-features-st_makevalid/
> It can be wrongly interpreted as a single
> ring polygon with self-intersection. However, it could be transformed into
> a valid OGC MultiPolygon with two parts ABF and BCDE.
correct.
--
sly (sylvain letuffe)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
> Drawing this as one outer ring touching itself in one point seems to > be invalid for JTS (self-intersection) while drawing an inner ring > touching outer ring at one point is OK for JTS. I don't know JTS, but as you describe it, it implements validity as define by the OGC for this case. See here, the "banana polygon" : http://workshops.opengeo.org/postgis-intro/validity.html But it shouldn't be forgotten that multipolygons in OSM are not OGC multipolygons. The sentence used on the wiki since the beginning was : "multipolygon relation can be used to build multipolygons in compliance with the OGC" In the case http://wiki.openstreetmap.org/wiki/Relation:multipolygon/validity#F_- _Polygon_self_touching_on_one_node A valid OGC can still be constructed as you described it (one outer and one inner touching on one point) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
> I don't think that the "touching inner ring" issue was ever limited to > "two or more consecutive points". Your page is the first time I read this. As far as I understand the OGC validity of multipolygon, the answer is yes. Because in the first place, OGC standard does not consider touching inner ring by one isolated node to be invalid. Therefore, the deviance to the standard as expressed in the sentence "with the notable exception of touching inner rings" is only true for "two or more consecutive points" (Like the example shown here : http://wiki.openstreetmap.org/wiki/File:Multipolygon_Illustration_8.png ) > Personally I don't consider the "8" shapes valid And : http://wiki.openstreetmap.org/wiki/File:Multipolygon_Illustration_touching_on_one_point.svg ? Your input is valuable ;-) But for a clarity's sake I think we have to answer those questions : a) Do we need touching outer polygons in OSM b) If yes, can it be allowed as single way in 8 shape or modeled as 2 ways touching at a single point c) If no, the long standing sentence "the multipolygon relation can be used to build multipolygons in compliance with the OGC Simple Feature standard" should be amended with other exceptions added My proposal for a) is yes we need them (I can show boundary examples on requests) and I don't care if b) is or isn't considered invalid as long as it is written on the wiki > - and I don't think > there's a difference between the one with a node in the middle and the > one without. There isn't if you focus on the OGC MULTIPOLYGON geometry built from this relation (it will be the same in the end), but it might be a computing overhead that we might want to forbid in the first place : at mappers side. > I don't consider the "inner ring touching outer ring in > single point" case to be valid either. I also have boundary examples on request to express it is needed, but we can either build those cases with that : http://wiki.openstreetmap.org/wiki/File:Multipolygon_Illustration_self_touching_on_one_point.svg -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
Le dimanche 14 octobre 2012 15:48:08, sly (sylvain letuffe) a écrit : > I'll come back later with examples as a graphic is far easier to understand > than fix font ASCII art Here is some food for thoughts : http://wiki.openstreetmap.org/wiki/Relation:multipolygon/validity Feel free to add other examples that you think need clarification. And of course comments are welcome. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
Le dimanche 14 octobre 2012 04:37:43, Willi a écrit : > Imho it's not only bad behavior to change a wiki page 19 times on the same > day it's harming OSM. Agreed. > Having the discussion on the OSM-dev list makes this > even worse. What about starting here, to reach GIS aware people, then present the final idea on talk (backed up by definition ranging from clear mathish un- understandable by common people definition to clear examples with graphics) ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] multipolygon relation and non-closed ways
Le samedi 13 octobre 2012 21:49:21, Frederik Ramm a écrit : > (Frankly I am surprised > that you thought the list was necessary, as I believed everything to be > explained properly already.) I'm not surprised, I've been asking for a while what validity means in the context of multipolygon relations in OSM (on tagging list), especially in strange border cases, but I'm still missing answers. I also guess that since changes on the wiki are recurrent about that, I'm not the only one unsure about what is and what isn't valid. But maybe I failed to reach the good list, and dev is better suited for that as the answer is far from trivial, and is indeded more complex that you seam to imply (+include some GIS background needs). But nothing is ever too late isn't it ? I'll come back later with examples as a graphic is far easier to understand than fix font ASCII art As a starting point for the shorter definition of what validity is, I suggest referring to the OGC standard this way : ''' A multipolygon relation in OSM is considered valid if it can be used, without discarding nodes or ways or part of ways to build a valid geometry as define by the OGC Simple Feature standard (http://www.opengeospatial.org/standards/sfs) with the notable exception of touching inner rings (see below). ''' -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Streaming Replication
> Hi All, Hi Brett, > To solve this, a new streaming replication mechanism has been developed. > Under the covers the same database queries are utilised, but the process > performing the queries runs continously and polls the database for changes > at a shorter interval. Nice feature, it might not be of interest to everyone but people operating a ro mirror for editing purpose might have interest. And that's my case, so thanks for that ! > However, I'd love to see it get > some usage and would welcome any feedback. Here I am allready ! I've plugged that to my test server's overpass db updates and things seams to work as expected. Let's see how it performs in case of network hickups note : I have a strange issue with all objects's timestamp, they seam to be shifted by -1 hour -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
> Building such a meta > platform is not an easy task because "bugs" are much more than just > popup markers on the map - bugs can also be a way, or a collection of > ways, or a convex hull around a broken polygon; a good platform will > also allow the user to flag those bugs which are false positives. All > this in an environment where a multitude of QA providers feed hundreds > of thousands of check results into the platform every day. For your information, osmose http://wiki.openstreetmap.org/wiki/Osmose (free software) developed for and by the french community is doing kind of exactly that. A meta plateforme taking inputs from distant plugins as xml files, false positive management, plugin framework (optionnal), redistibution API to embeded devices, a JOSM plugin, allready an OpenstreetBugs layer and a quick editor. It lacks however the convex hull around buggy areas. We don't really have the server ressources (yet) to make it available world wide, but the software beeing GPL, if I remember correctly, could be re-used and upgraded to run at larger scale. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Osmosis - Cutting out a bounding box from a diff file (.osc)
hi, > but is it a bug? or am I missing the good reason why there is this factor > 100? I searched the source code to find the reason why there is a x100 on coordinate, but couldn't find where it's done. But my guess is that for performance reasons, the field to store x/y was chosen to be an integer instead of a float. And because rounding a spherical mercator to the unit would loose sub meter precision, it was stored as multiplied by 100 for ~cm accuracy -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Osmosis - Cutting out a bounding box from a diff file (.osc)
> I just tried another hack, but feel it's going nowhere: > >1. Export all from the database to an osm file => full.osm That's not possible using an osm2pgsql schema with existing tools I know. > Although a bit heavy I thought it would work... But apparently, Osmosis > cannot read the PostGIS database, if using the osm2pgsql schema, right? correct. osmosis can read a... osmosis schema ;-) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Osmosis - Cutting out a bounding box from a diff file (.osc)
> I will end up > with the full planet (as every single node in OSM might be modified one day > or the other...) Well, My guess is that it will take a lng time, so long before that you might need to re-import your extract for whatever fail delete, style change, software update, and so on. Thus returning your database storage back to your extract's size only. Note that a regular extract re-import is also a way to clean what is outside your zone of interest. > By the way, would this be a bug or a request of improvement for osm2pgsql > or osmosis? I think it is not possible to do it properly, a dirty hack loosing data from time to time could do but that will still require server ressources. As said earlier here : http://lists.openstreetmap.org/pipermail/dev/2012-August/025503.html A share zone for regional diffs exports could do, or support for the new augmented diffs in those tools. But both still have to be done. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Osmosis - Cutting out a bounding box from a diff file (.osc)
> But it looks like I have a problem now... > >1. I need to keep the tables ways, rels and nodes because of the diff >updates >2. Those 3 tables will continue growing up with each new diff, until I >reach the storage capacity of the server > > The only solution I see is to increase the storage capacity ;-) > is a post-processing that will erase from PostGIS > all the nodes lying outside my bounding box + all the ways using one of > thses nodes + all the relations using one of thses nodes or one of these > ways. Doing that will erase data that you want in your bbox, suppose a way crossing the border of your bbox, it will be erased while it still had nodes in the bbox. A better but longer solution is to remove every ways with no nodes in the bbox. But that probably won't be as easy as it sounds. Better increase you storage capacity ;-) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Osmosis - Cutting out a bounding box from a diff file (.osc)
On mercredi 5 septembre 2012, Stéphane Henriod wrote: > Hi again yo, again, > Does this mean that I can simply empty them regularly? You can, but then applying diffs won't be possible at all. Those table are not only used to reduce memory usage of the first import, but also to hold all data to solve the problem I mentionned about diffs not having all informations of their constituent nodes in order to apply them. > It looks to me like > Mapnik won't need them at all, but I might be wrong? It depends on your mapnik style sheets, but my guess is that there are no reasons to use those table for rendering (They don't hold geometries) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis - Cutting out a bounding box from a diff file (.osc)
On mercredi 5 septembre 2012, Stéphane Henriod wrote: > Dear all Dear stéphane, > My problem is that, despite the *osm2pgsql -b* option, it seems that my > bounding box is ignored and the full diff file is added to my database, As > a consequence, the database is growing very quickly and includes lots of > data outside of my area of interest. I can confirm this behaviour of osm2pgsql, although I haven't digged in too much to see what this "-b" switch does in regard to diffs (if it does anything at all) I't working for the first import, but diffs seams to be applied for the whole world. Your finding about the table sizes seams that of slim tables planet_osm_rels/ways and I suppose nodes as well are filled with data out of the bbox. > My question is thus: is it possible to cut out the bounding box from the > diff file before calling osm2pgsql? I would say : not correctly unless you have a full database of the world somewhere > If I understand correctly, the > bounding-box task ( > http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--bounding-box_.28--bb.29) > only works on planet files, not diff files. Correct? Correct. > Is there a way to get the following process working: > >1. Get the last hourly diff file >2. Extract from the diff file all the objects within my bounding box >3. Apply this to the PostGIS database with osm2pgsql My guess is that "no", unless you have a database of the world. The reason for that, (that's a guess) is that diffs (.osc) produced do not necessarly have geographic informations with objects other than modified and added nodes. In a diff you can have changes about a way without any of it's constituent nodes (for instance, when the change only consist of tag changes, when the ways is changed to use other allready existing nodes, etc.) This means that the diff is not enough to tell wether the modified way/relation is or isn't in you bbox, unless you have a world database to chech where it's nodes are. However, the new introduction of "augmented diffs" : http://lists.openstreetmap.org/pipermail/dev/2012-August/025487.html might help doing what you want, but AFAIK, no tools can consume them yet. Other option, is, depending of you bbox of interest, regional diffs from here : http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Regionaly_limited_diffs -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Regional Diffs
Le mercredi 29 août 2012 21:22:45, Christian Quest a écrit : > Have a look at http://download.openstreetmap.fr/redaction-period/ and > give you feedback as it is a bit experimental. I updated to wiki to reflect this "probably more permanent" URL, but this is the same service as the one Markus was refering to (just a different url) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Regional Diffs
Le mercredi 29 août 2012 20:38:44, mar...@gmx.eu a écrit : > Hi, Hi, > I just saw this new section in OSM Wiki: > http://wiki.openstreetmap.org/w/index.php?title=Planet.osm%2Fdiffs&action=h > istorysubmit&diff=801359&oldid=792156#Regionaly_limited_diffs I thought it could be usefull to some people without powerfull enough hardware who only want regional db extracts. > Probably, daily or hourly diffs would suffice for most users. > Does anyone know if this kind of diffs is already available somewhere? We search but couldn't find any services, so we went the path to build are own workflow to produce them > If not, is there a market for this? At least for one database (europe) I operate and one database (France) the french community uses for analysis : yes. That's why we only provide the diffs we need. ps: note that this is not related at all with new augmented diffs anounced recently and only a coincidence. But those "augmented diffs" might be another alternative to keep a regional db up to date. -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql: super slow relations
Le mardi 17 juillet 2012 16:03:55, Ramas a écrit : > Hi, > after some updates in my system osm2pgsql relation processing in slim mode > become very slow: What's values are your fsync and synchronous_commit settings in postgresql.conf ? With fsync = on I've seen the same behaviour as yours, try turning it off to see if speed is increased ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] (Multi)Polygon handling
> But we need to agree on what is valid and what is > treated as invalid. I think this is the first step we should start with (or at least define it somewhere with words while building test cases). Test cases are good technical tools to automatically evaluate algorithms and their multi-polygon compliance, but we should define, for the mappers, what is invalid and valid. What we have here : http://wiki.openstreetmap.org/wiki/Multipolygon is : "Generally, the multipolygon relation can be used to build multipolygons in compliance with the OGC Simple Feature standard (http://www.opengeospatial.org/standards/sfs). Anything that is not a valid multipolygon according to this standard (e.g., polygons with intersecting rings) should also be considered an invalid multipolygon relation, with the notable exception of touching inner rings (see below). " But I raised concerns about the exact meaning of this sentence here : http://lists.openstreetmap.org/pipermail/dev/2012-May/024948.html -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql - COPY_END SSL error?
> I chose a bounding box that is only slightly larger than Hungary - so > that the content doesn't stop at the national border. while importing > hungary.osm.bz2 is very fast (less then an hour), importing this > bounding box via europe.osm.bz2 takes very very long Weird, do you mean your import allready ran for ~30 hours or so ? My guess would be that your bbox setting wasn't applied at all and what is happenning is that you are importing the whole europe. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql - COPY_END SSL error?
On mardi 29 mai 2012, Ákos Maróy wrote: > All indexes on planet_osm_polygon created in 577s > Completed planet_osm_polygon > Stopped table: planet_osm_rels in 819s You can find here a typicall session of import with osm2pgsql and compare with your own to have a rough estimate of how long it will continue : http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks/Dell_R610_import_1 > I wonder what it is doing all this time.. building indexes mostly -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM Multipolygons invalidity
Hi, I've asked this question on the tagging list, but I guess this wasn't the best place to talk about it since it involves a question of convertion ability for algorithms to go from OSM's definition of multipolygons (MP) to the OGC simple feature MP entity. I consider the wiki, as a source of information about tagging, to be the main ressource of information to answer "Am I allowed to tag a MP this way or not" To sum it up, the wiki says that a OSM MP should be considered valid if "the multipolygon relation can be used to build multipolygons in compliance with the OGC Simple Feature standard" (with the known deviation of touching inner rings) However, this is a bit confusing for me because this sentence does say "can be used" but I guess with a powerfull algorithm, it is always possible to build a OGC MP from any OSM MP (possibly by droping unclosed ways, re-cutting ways, etc.) even if ending with a empty OGC MP after droping everything (which is valid according to OGC) My guess is that this sentence implies (without saying it) things like if a member way need to be dropped, then it is an invalid MP. Is that so ? Real border case of validity : http://wiki.openstreetmap.org/wiki/Talk:Relation:multipolygon#special_case_of_touching_outer_.28or_inner.29_rings_on_one_or_more_points_only I didn't thought of this case at random but because a well known algorithm used around here (osm2pgsql) doesn't manage to build a valid OGC MP from such a case. But I'm not talking about it to blame osm2pgsql but to ask if complexity arising from such cases should be considered too much for an algorithm, and pushing back the problem to the mapper saying "no, we consider it invalid for OSM please don't do it." -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] minute dumps
On mercredi 4 avril 2012, Frederik Ramm wrote: > Hi, (...) Thanks. That's all good news for data consumers -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] minute dumps
Hi, > [1] They'll include changes by the background cleaning script so the > diffs will be larger than usual. This is interesting, is there any blog post/wiki page explaning this ? (I read : http://blog.osmfoundation.org/2012/04/04/api-read-write-returns/ ) Sorry if I missed it, but is the redaction process ran as any ususal edit process ? is there an identified account all redaction is done with ? Can I assume, that, after consumtion of all redacted diffs (in an osm2pgsql like scenario, ie : no history kept) my database only contains odbl licenced data in the end ? Again, if all that is answered somewhere I missed, could you give me a pointer to it ? -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql diff imports benchmarks
> > gis=> explain select id from planet_osm_ways where pending; > > Index Scan using planet_osm_ways_idx on planet_osm_ways > > (cost=0.00..916513.78 rows=43724 width=4) This is what I've got after a re-index : gis=# explain select id from planet_osm_ways where pending; QUERY PLAN -- Index Scan using planet_osm_ways_idx on planet_osm_ways (cost=0.00..1157.47 rows=1 width=4) (1 ligne) and that saves me around ~20 seconds per minute diff -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql diff imports benchmarks
Hi, In the process of trying to speed up diff imports with osm2pgsql, I'm searching for a "typical" output of osm2pgsql importing one minute. I know of : http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks But beside the one I just wrote, there are no output of what is "expected" for a one minute (or few minutes) diff import. Could someone operating and maintaining a planet osm2pgsql database be so kind as to provide me one (or more in order to have average metrics) ? On a side note, while loging long queries I ran into a weird long one about retrieving pending ways during diff import. It turned out that the index on the "pending" field wasn't used at all and led to a full scan of the planet_osm_ways table, increasing noticeably the diff import. I don't feel I have imported data in a weird way and don't know if others are affected by that, but you can check if you are affected by runing : "select id from planet_osm_way where pending" which should run in less than a second if index is well used. To solve the problem post import, we did re-index and ran analyse reindex index planet_osm_ways_idx; analyse; -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Nested relations and generic implementation of geometry constructs - Was :osm2pgsql patch for nested relations
> type=boundary is much the same as type=multipolygon. I have always argued > that only type=multipolygon should be used for boundary=administrative. One > down. I do agree with you on the fact that boundaries "could" or "should" be tagged as type=multipolygon, but that is not the case in the database. More than that, there are members with role=subarea in it making them no more a geometry valid as defined by type=multipolygon > > type=waterway --> build-graph special to handle the main stream and the > > side streams to build a graph and not a linestring (special handling at > > GIS side because it doesn't exist as valid WKT) > > Never heard of those. There are only about 3500 cases in the database and no > documentation in the wiki. Another one down. wiki page is here : http://wiki.openstreetmap.org/wiki/Relations/Proposed/Waterway -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Nested relations and generic implementation of geometry constructs - Was :osm2pgsql patch for nested relations
On vendredi 2 mars 2012, Peter Körner wrote: > By defining algorithms to work with relations in one place and mapping > relation-types to that algorithms in another place. The idea looks sexy, but relations have so many different construction that you will probably end with as much algorithms as there are relation types. > For example: Let's see what exists : type=multipolygon --> build-polygon (but waterway=riverbank and some lakes are constructed with touching outer rings loosing the true geometry, if proper processing is needed we whould need special tweaks of those cases) type=multipolygon are note expressly said to allow nested relations type=boundary --> almost build-polygon, but some members sneaked their way as subareas while not part of the geometry and roles are different. In those there are cases where nested relation are forming the geometry type=site (relation become categories ?)--> build-polygon but only for members with role=perimeter and tags are pushed down type=route --> build-linestring but only one route/(blank), and conditionnal on those backward/forward/north/south/east/west type=waterway --> build-graph special to handle the main stream and the side streams to build a graph and not a linestring (special handling at GIS side because it doesn't exist as valid WKT) type=street/associatedStreet --> build-linestring with role conditions There are others, but each might need some special roles conditions and the way tags apply to what might be kind of special > The goal should be reducing the number of algorithms and re-using them > as much as possible. I understand the goal, but fear that it won't be so easy -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch for nested relations
> their member ways are just added to the planet_osm_lines > table, they are not even concatenated. > Sven Are you sure ? I might be mistaken, but thought they were concatenated and then split again if in exces of 1°x1° (line ~1080 in output-pgsql.c ) I haven't fully checked what does the build_geometry function however, but I do remember that I got full geometries if I changed the split_at to 360° -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Nested relations and generic implementation of geometry constructs - Was :osm2pgsql patch for nested relations
> I know that there are precedents, but I am highly averse to coding > specific relation types into osm2pgsql. So do I, as you mentionned below, the problem is not osm2pgsql specific, but will need work in every tool to maintain support for ever changing relation's way of tagging, and every new relation types making it hard to have generic processing. Unfortunelty, that problems comes from what's in the database and the fact that several relation proposals exists with different meannings and different tagging structures. By "meannings" I mean that some relations are made to build a bigger geometry (type=multipolygon is one), some to records facts unreleated to geometry (i.e type=restriction ) The question is how do we improve that, while keeping free tagging system ? We are having a discussion about type=waterway relation here : http://wiki.openstreetmap.org/wiki/Talk:Relations/Proposed/Waterway in the generic vs specific relation constructs. Those favoring specific relation tagging have raised valid concerns about how hard it will be for mappers to enter nested relations and/or generic geometry building relations. But as a data consumer and algorithm maker, it might become a nightmare to support all types of roles and logic behind those. > one is pushing the geometry > up, the other is pushing the tags down. I guess both would be needed depending on what was the purpose of the relation. When the parent relation doesn't describe a geomety, but is a way to factor tags (name, ref, ...) maybe we need to push the tags down. When the parent relation makes a big geometry made of several child relation having a part of the geometry, that's the opposite. (country boundaries made of several linear boundaries comes to mind type=boundary_segment) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql patch for nested relations
Le jeudi 1 mars 2012 11:30:03, Sven Geggus a écrit : > You could just skip the code following "if( strcmp(type, "route") == 0 )" > in output-pgsql.c, but I doubt that this will increase your processing > speed significantly because most of the hard processing work is for > multipolygons relations anyway. Reading that thread, I remembered that I don't need type=route either, I've commented that part of osm2pgsql but the speed gain is hardly noticeable (if at all) -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql not importing routes?
Le lundi 27 février 2012 20:25:02, yvecai a écrit : > Then I installed osm2pgsql 0.66 from ubuntu repo and it works. > Any luck to have an up to date osm2pgsql in next Ubuntu LTS ? Are you using the same style file (-S ./style) with the two versions or are you using the default style for each ? I just checked two osm2pgsql db I have access to (both were built with the same 2 month old osm2pgsql version) the first as the relation you are searching for while the other hasn't. The one having it has this in the style file : node,way routetext linear In the other it's commented out (on purpose) aren't you missing this line ? -- sly (sylvain letuffe) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting information from relations.
> It looks like osm2pgsql does not merge route members into selfstanding > geometry objects. Therefore the following sample queries will send routes > as a short segments. In case you are interested, I have the patch to solve the case, but it's really easy to implement in osm. Find // Split long linear ways after around 1 degree or 100km (polygons not effected) in output-pgsql.c and change the value to something huge, here you are -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] traces, export, download and update
Hi,
As well as the original poster, I might be interested in raw gpx tracks data
> > The plan is just to provide a dump - if other people want to build clever
> > tools on top of that data then that's fine.
It remind me of that thread :
http://lists.openstreetmap.org/pipermail/dev/2009-December/017802.html
But since old tracks don't get updated, since many people might just be ok
with a simple tar.gz of all public visible gpx files (they'll do the fancy
formating, database inserting,...)
Couldn't we just get them once with the API by ID ? and put it somewhere for
sharing?
Nothing fancy or complex
Some kind of script shell I just tried :
for x in `seq 1 1162147` ; do
wget http://${auth}@www.openstreetmap.org/api/0.6/gpx/$x/details -O $x-
details.osm
wget http://${auth}@www.openstreetmap.org/api/0.6/gpx/$x/data -O $x.gpx
done
some clean up and a tar and hop
One at a time, no multi-request, a quick test on 100 shows me in 3 day it's
over
I'm willing to do it myself and share the result if it doesn't look too much a
over load for the API
== head buried in the sand to avoid the sword ==
--
sly (sylvain letuffe)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] 0.6 API read load balancer/proxy prototype with lower limits
Hi, Following : http://lists.openstreetmap.org/pipermail/dev/2011-December/023945.html http://lists.openstreetmap.org/pipermail/dev/2011-December/023981.html I've set up a read load balancer/proxy to the main OSM API Better than a thousand words, here it is : http://beta.letuffe.org/api/ It should be usable wherever any osm editors, tools, bulk downloader use http://api.openstreetmap.org/api - For example, you can put this adresse in JOSM F12 -> connexion -> API url - It only supports the France area for now (it's a demo) - You need valid credential at the 0.6 API dev server : api06.dev.openstreetmap.org if you want write operations to work In a few words "why" : The main API server is applying sanity restrictions in order to limit per IP, per area or per elapsed time calls sent to it. This simple proxy is reducing such calls by answering the data request calls itself. It's only a prototype showing one solution among many others to accelerate read calls and reduce load on the main API server. In a few words "how" : When handeling a request, any of map, node, way, relation, nodes, ways, relations and capabilities GET calls are converted to the overpass API syntaxe and forwarded to a local (or even could do to a distant) overpass API server. When handeling any other PUT, POST, DELETE, WHATEVER or GET requests, it forwards the call to the 0.6 API dev server acting as a transparent proxy as much as possible. Advantage beeing that no modifications are needed in the client if it supports the 0.6 API (beside changing the target URL of course) note: Your credentials are clear text transmitted to my server, but no storing is done (you'll have to trust me or use unimportant credential from de dev server) note: the dev server does not have a live copy of the main osm API, so any modification done to existing data will fail with a version error (you have to create fresh data to test it) note: please DO NOT try to force re-upload to the live API unless you are very sure of what your datas looks like, because I suspect a few bugs to still be hidding around -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Hi, > We're working > hard on getting the relevant hardware in place to start trialling this > out, but it's a big project. Many thanks for the insight > The original topic was about replication for rendering, so a comment > on that Whooops, I haven't read that thread far enough in the past to discover I'm off topic. I do agree that livre replication for rendering has few to no interests, and I was talking about live replication in order to load balance editing read calls. I'll start a new topic about that later but we are in the process of acquiring 3 huge servers for our activities in France, one of wich might be used for API ro calls and I'll conduct a few tests to check if a 3 minutes lag behing main db doesn't have significant edits drawbacks. Final goal is having faster and less limited ro api for editing -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Re-Hi, > > Exemples of bad requests are : > > http://www.openstreetmap.org/api/0.6/relation/1403916/full > > That looks like something I need to investigate. My guess is that it is > hitting the timeout but it should be reporting that better. I guess you guessed it well, there is just "too much" (to the extent of what "too" means in this case) data in this relation, and the /full calls push it to hit a well placed timeout > Well that's not an error, it's a deliberate limitation of the API. (...) > Again that is a deliberation limitation of the API to converse resources > and share them fairly. I did not mentionne anywhere those were errors. I do know it is intentional, I stated it in my previous email. What I'm discussing and exploring is the possible existence of improvement to satisfy currently "bad requests" still usefull in the mapper's editing process. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
> Strangely, if you encounter a problem with the server then it is > necessary to ACTUALLY TELL SOMEBODY ABOUT IT. Everybody be cool, this is (not) a robbery ! Nor any disregard to actual sys admin/dev of the main api server doing an amazing job. This is just a general statement concerning isolated cases (which I didn't call "problems" due to needed and intended protections against what some people use to call "bad requests") What I'm considering is thinking about moving those so called "bad requests" to new category like "burden another server please" Exemples of bad requests are : http://www.openstreetmap.org/api/0.6/relation/1403916/full http://api.openstreetmap.org/api/0.6/map?bbox=2.253,48.8145507,2.430,48.901 And more globaly any request that requests a lot of data Other, more subtle "bad requests" due to what seams collateral damage like : http://lists.openstreetmap.org/pipermail/dev/2011-December/023945.html There are maybe other cases I don't know about as I'm not a main api server sys admin, but I try to guess, and maybe you could help me by answering this question : Will the maintenance and stress on this server (therefore on it's sys admins) whould be painless if all read api calls were directed to other servers ? -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Hi, On jeudi 15 décembre 2011, Frederik Ramm wrote: > But: Anyone who really wants to, and has the resources to, can set up a > full database today, feed it with minutely diffs through Osmosis That is true, but there are no solutions for bellow minute synchronisation (like near real time synchronisation) Altough the need for real time synchronisation is limited, as a mapper I face, sometimes, main API slowdowns, http 500 answers and other irritating conditions. Almost always in retrieving operations, not writing operations. Maybe, and only "maybe", one or two ro api servers could help in those cases Maybe, still "maybe", a 2 or 3 minutes behind ro api could do the job but I don't know if that whould cause risks of editing an object of the past. -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slow slim import
> Kai has suggested to me changing configuration of postgresql like this: > > fsync = off > synchronous_commit = off > > And this worked like a charm, now I'm getting significantly faster > imports compared to old config and old osm2pgsql. Whaoo ! It "just work" (and very well) -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slow slim import
> Can you try if using the switch --number-processes=1? That should disable > any parallelisation and make it more or less the same as before. (Although > there is a slight difference in transaction handling) It's x2 faster with --number-processes=4 instead of 1 but still far from older version > Otherwise, can you check if you are getting 100% hit ratio from the node > cache? yes, 100% extracted output : $ osm2pgsql --number-processes=4 -C 2000 -s -S default.style -m -d gis2 monaco.osm.bz2 Using 4 helper-processes processing way (0k) at 0.00k/s All child processes exited Pending ways took 6s at a rate of 158.17/s node cache: stored: 10409(100.00%), storage efficiency: 62.61% (dense blocks: 1, sparse nodes: 10401), hit rate: 100.00% Osm2pgsql took 9s overall $ osm2pgsql --number-processes=1 -C 2000 -s -S default.style -m -d gis2 monaco.osm.bz2 Using 1 helper-processes processing way (0k) at 0.00k/s All child processes exited Pending ways took 15s at a rate of 63.27/s node cache: stored: 10409(100.00%), storage efficiency: 62.61% (dense blocks: 1, sparse nodes: 10401), hit rate: 100.00% Osm2pgsql took 18s overall (using revision 26892) $ osm2pgsql -C 2000 -s -S default.style -m -d gis2 monaco.osm.bz2 Going over pending ways processing way (0k) at 0.00k/s Going over pending relations node cache: stored: 10409(100.00%), storage efficiency: 2.32%, hit rate: 100.00% Osm2pgsql took 3s overall The old version didn't display time spent in processing way, but for such a small extract it's a few ms -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tips to reduce bulk tiles downloads ?
> From the download pattern. Humans have a linear distribution of tile > requests with most in zoom 14 and 15. Tile scrapers show an exponential > increase with higher zoom levels. whaoo ! That shows you've gone far in tile scrapers activity analyse and it soon logical about "normal" human behavior. Are you using mod_tile ? does it permit all that ? have you an other queue management/penalty/rate limiting software ? > > file:///Applications/Install/83F78CDD-FB29-E011-854C-00237DE2DB9E/Install/ > > and user agent is : NativeHost > > > > file: referrers hints at a local test installation somewhere. I had a private email from someone from the list who tried to search google for the strange "83F78CDD-FB29-E011-854C-00237DE2DB9E" string and found the culprit. That's a windows phone apps claiming to provide "off live browsing" (wich I suppose means tile scraping ;-) ) for a lot of map providers among which mine is listed > > Then comes (or alike): > > "Dalvik/1.2.0 (Linux; U; Android 2.2.2; LS670 Build/FRG83G)" > > > > The first looks fake, the second does not ring a bell except that it's > likely something on Android. But a per zoom level limit should help. After diging, il looks like a few android apps do implement tile scraping while other do implement only human browsing. I found that mytrails as my renderer listed and also has a bulk scraping menu > That seems rather random - I am having the most trouble with an Android app. You are right, I just found one ;-) -- sly qui suis-je : http://sly.letuffe.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tips to reduce bulk tiles downloads ?
> If you were to use mod_tile, that would already be in there; mod_tile > has two different delay pools, one for tiles in cache and one for tiles > that need to be rendered. For historical reasons I didn't start with mod_tile, but given those new functionnalities, I'll have a try with it. -- sly qui suis-je : http://sly.letuffe.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tips to reduce bulk tiles downloads ?
On mercredi 27 juillet 2011, NopMap wrote: > You know about the throttling of download speed in mod_tile, I guess. Well no, because I don't use mod_tile, but that is a good idea, instead of blocking completly the ability to download, reducing/limiting the hit rate per minute per IP might be a good idea. I'll try to do something about that. > On my server I use a tile limit per zoom level (automated tile scrapers show > an exponential increase with higher zoom levels) I plan to add a "tile not in the cache penalty" because that's what put all the load, deliverying tiles in cache as close to no impact > and a penalty for > missing/fake agents. How do you know what are fake agents ? > Excessive IPs are blocked for 24h. That's what I'm currently doing, but that's not perfect > Actually, a good strategy is to try and identify the application causing the > load and talk to the author. That sound a good way to find compromise, I don't want to ban, I just want my server not to be sluggish like snails for legitimate usage, but I have problem identifying applications : > Many applications do have a user agent. > Currently the most frequently blocked downloaders in my log are MOBAC and > Locus. > > Do you have any idea who is causing the load? By far the one with more hits (before I rate limited hits) looks strange : It's refererer is : file:///Applications/Install/83F78CDD-FB29-E011-854C-00237DE2DB9E/Install/ and user agent is : NativeHost The second one has what looks a fake user-agent : "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)" (sequential download and IE 6.0, seriously ;-) ) Then comes (or alike): "Dalvik/1.2.0 (Linux; U; Android 2.2.2; LS670 Build/FRG83G)" But android apps seams more friendly with my server as I didn't find what looks bulk dowloads. android devs more respectfull ? ;-) I don't know what MOBAC is but I only found 1000 hits with : "MOBAC/1.9_beta_6" in 6 days, so that's clearly not a problem for me. -- sly qui suis-je : http://sly.letuffe.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Postgres 8.4 performance trouble in osm2pgsql setup
(sorry for breaking the thread) Just a mail to give other results in experiments with postgres 8.4, I'm in the progress of migrating from debian lenny (postgres 8.3) to squeeze (postgres 8.4) and I'm of concerns about initial import speed with osm2pgsql and later diffs. I was just going to do it blindly but after reading you'r mail I gave it a try, and haven't run into the speed regression you mentionned. Test case : wget http://download.geofabrik.de/osm/europe/france/alsace.osm.bz2 time osm2pgsql -C 1200 -s -S ./default.style -G -x -m -d gis ./alsace.osm.bz2 (...) First server (squeeze postgres 8.4, osm2pgsql SVN version 0.69-exported) real4m33.505s Second one (lenny postgres 8.3, osm2pgsql SVN version 0.69-exported) real5m4.108s Unfortunately both server are not equivalent, so the test is not perfect, but I would say they are closed in configuration for such a test (First has a RAID0 array and 2GO RAM while the other as a RAID1 array and 32Go RAM) Mermory doesn't seams relevant in this test case, so the 37s diff might just come from the RAID0. But I haven't reproduced you'r "4 time slower" The only tweak in the default configuration provided by both lenny and squeeze is : shared_buffers = 128MB -- sly Sylvain Letuffe sylv...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql / diff / upercase tag
Hi, Maybe this case is known (also a google search didn't pointed me to it) but osm2pgsql diff adding mode fails when there is a tag with upercase caracters Sorry if that bug report is in the wrong place : postg...@binael:/home/postgresql $ svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/ (...) postg...@binael:/home/postgresql $ cd osm2pgsql postg...@binael:/home/postgresql $ make (...) postg...@binael:/home/postgresql $ cat test.style node,way IN text linear postg...@binael:/home/postgresql $ ./osm2pgsql/osm2pgsql -s -S ./test.style -d not_osm /tmp/test.osm osm2pgsql SVN version 0.67-18306 Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point Setting up table: planet_osm_line Setting up table: planet_osm_polygon Setting up table: planet_osm_roads Mid: pgsql, scale=100, cache=800MB, maxblocks=102401*8192 Setting up table: planet_osm_nodes NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes" Setting up table: planet_osm_ways NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways" Setting up table: planet_osm_rels NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels" Reading in file: /tmp/test.osm WARNING: Found Out of order node 298786602 (2388935,810) - this will impact the cache efficiency Processing: Node(0k) Way(0k) Relation(0k) Node stats: total(219), max(452930789) Way stats: total(5), max(38344957) Relation stats: total(0), max(0) Going over pending ways Going over pending relations Committing transaction for planet_osm_line Committing transaction for planet_osm_roads node cache: stored: 111(50.68%), storage efficiency: 1.35%, hit rate: nan% Stopping table: planet_osm_nodes Committing transaction for planet_osm_point Stopping table: planet_osm_ways Stopping table: planet_osm_rels Sorting data and creating indexes for planet_osm_line Sorting data and creating indexes for planet_osm_roads Committing transaction for planet_osm_polygon Building index on table: planet_osm_rels Sorting data and creating indexes for planet_osm_point Building index on table: planet_osm_ways Stopped table: planet_osm_rels Stopped table: planet_osm_ways Sorting data and creating indexes for planet_osm_polygon Stopped table: planet_osm_nodes Completed planet_osm_roads Completed planet_osm_point Completed planet_osm_polygon Completed planet_osm_line postg...@binael:/home/postgresql $ ./osm2pgsql/osm2pgsql -a -s -S ./test.style -d not_osm 200910251215-200910251216.osc.gz osm2pgsql SVN version 0.67-18306 Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point Adding new column "IN" to "planet_osm_point" ALTER TABLE planet_osm_point ADD COLUMN "IN" text; failed: ERROR: column "IN" of relation "planet_osm_point" already exists Error occurred, cleaning up -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] mapnik rendering very long on low zoom
Hi, For different reasons, I need re-building a whole lot of rather old (~8month) mapnik tiles across europe I had. I am in the process of building it from zoom 0 to zoom 12 (other levels are handeled with on demand cache system ) >From level 0 to 5 things are going quite fast (presumably because there is almost nothing on it) but zooms ~6 to 8 are generated terribly slowly. 9 to 12 being still quite slow and 13 or less starts to be manageable. I'm using the generate_tiles.py program and a one year old mapnik file to trouble shoot the problem (the same I used 8 month ago) and I can't remember this process to be that slow ! The machine seams io-bound, top : Cpu(s): 4.8%us, 1.8%sy, 0.0%ni, 69.4%id, 24.0%wa, 0.0%hi, 0.0%si, 0.0%st load 1.00, iostats tels me the reading speed on disks is around 2MB/s (~100 times less than my burst disk/raid0 speed) swap isn't used I've tweaked postgres with the wiki recommanded settings and a command like this : $ time ./nik2img.py -m osm_copy.xml -i jpeg -o test.jpeg -s 500,500 -e 2,40,8,46 (...) real5m24.251s user0m5.684s sys 0m0.572s poorly generates a small image in 5 minutes ! (dual core 2.6GHZ /2GO memory ) Is my memory faulty on what it used to be ? is this "expected" ? (or has data increase by a factor 10 ?) -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql bug on type=boundary/type=multipolygon ?
Hi, Sorry in advance if I'm in the wrong place (please point me to the bug tracking system if there's one) I might have found a bug in the current osm2pgsql code that handle boundary relations, but I'm not smart enough to kick his ass. Happens when : - handling type=boundary/multipolygon relations (with advance multipolygon sytem having more than one outer ring) - During diff append mode - To reproduce : Moving juste one node of one way and the POLYGON doesn't get updated Details : 1) osm2pgsql SVN version 0.66-15819M 2) Creating the 2 outer ways+relation $ osm2pgsql -s -S ./default.style -l -d not_osm creation.osc 3) not_osm=# select astext(way),name from planet_osm_polygon; select astext(way),name from planet_osm_line; astext | name +-- POLYGON((0 4,2 2,1 1,0 4)) | test (1 row) astext| name -+-- LINESTRING(1 1,0 4) | LINESTRING(1 1,2 2,0 4) | LINESTRING(0 4,1 1,2 2,0 4) | test (3 rows) No problem, except duplicates, but I suppose it's a feature 4) Moving one node (0 4) to (0 5): $ osm2pgsql -a -s -S ./default.style -l -d not_osm move_one_node.osc not_osm=# select astext(way),name from planet_osm_polygon; select astext(way),name from planet_osm_line; astext | name +-- POLYGON((0 4,2 2,1 1,0 4)) | test (1 row) astext| name -+-- LINESTRING(0 4,1 1,2 2,0 4) | test LINESTRING(1 1,2 2,0 5) | LINESTRING(1 1,0 5) | (3 rows) Both individual way got updated, but the relation and it's linestring representation wasn't All the same with type=multipolygon : not_osm=# select astext(way),name from planet_osm_polygon; select astext(way),name from planet_osm_roads; astext | name +-- POLYGON((0 4,2 2,1 1,0 4)) | test (1 row) astext | name -+-- LINESTRING(1 1,2 2,0 5) | LINESTRING(1 1,0 5) | (2 rows) The linestring representation of the relation is not there, but it's polygon version in planet_osm_polygon is not updated Thanks for reading, and/or pointing me to the good place -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Procedure to get a dev.openstreetmap.org account
> > (looks like my osm account is refused, because I suppose there is no sync) > > > Just create a account like you would do on osm.org Argh, what a ** I am. I thought the http://api06.dev.openstreetmap.org/ page was just a frontend to the real osm DB actions (beside API) since there was some rendering and the dev db looks empty. Sorry for the useless noise. Every thing is working -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Procedure to get a dev.openstreetmap.org account
On Wednesday 10 June 2009 13:12, you wrote: > Hi, > > The following test servers are available for testing of scripts, > rather than using the main OSM server: > http://apis.dev.openstreetmap.org/ > api06 or new06 are recommended. > It is better for you to use one of these ones, which are maintained, > rather than creating yet another api server on dev. Thanks for your answer, I made some tries with : http://api06.dev.openstreetmap.org/api/ Looks good, but who do I ask for an account to make some upload ? (looks like my osm account is refused, because I suppose there is no sync) -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Procedure to get a dev.openstreetmap.org account
Hi, In the process of trying to bulk import data to OSM for http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover (Dataset of france) We'd like to validate our procedure without the pain of installing and configuring a API 0.6 test server Is dev.openstreetmap.org a server intended to be usable for bulk upload tests ? and if yes, who could create an account for us ? -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minute diff breakage
Hi, I've read this thread : http://lists.openstreetmap.org/pipermail/dev/2009-May/015310.html about the minute diff from http://planet.openstreetmap.org/minute/ lacking some data, but I didn't figured out what the result was, maybe something like : "no solution on short term" ? I suppose it is advised to keep in sync with delayed minute diff : http://planet.openstreetmap.org/minute-slow/ instead ? Or is there any work around that I, as a client to the system, can do ? -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [OSM-talk] Osm2pgsql and column names with two underscores
On Monday 30 March 2009 14:08, Jukka Rahkonen wrote: > Hi, Hi, ( better transfert this to dev ?) > An osm2pgsql user writes on the forum about importing special tags > into PostGIS. > He has defined for example these tags: > node openGeoDB:telephone_area_code text > node openGeoDB:license_plate_code text I've read his problem, and proposed converting the osm input file with smaller tag name. The other solution might be to patch osm2pgsql. > Look at these: > "openGeoDB:telephone_are" a_code, > "openGeoDB:license_plate" _code > > It looks like column name gets truncated at the second > underline character. Coincidence ;-) Both tag name are also truncated at 24 chars exactly, let's see... > Could it be some bug in osm2pgsql, > or is it some other place where the error occurs? Looks like I went lucky : # grep 24 output-pgsql.c char buffer[1024]; char buffer[1024]; char osmtype[24]; char tag[24]; char datatype[24]; Recompiling osm2pgsql might help, if that limit of 24 is not present elsewhere -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] country boundary polygons
On Friday 27 March 2009 15:57, Andy Deakin wrote:
> Hi,
>
> I am looking for a simple answer to the question 'what country is
> lat+long in?', and for this I need the country borders as polygons.
For the technical part, I had extremely good and fast (~0.1s) result with one
single postgres/postGIS query :
echo "select osm_id,name,admin_level from planet_osm_polygon where way &&
transform('SRID=4030;POINT(6.0333 45.7666)',900913) and
_ST_Contains(way,transform('SRID=4030;POINT(6.0333 45.7666)',900913));" |
psql gis
osm_id | name | admin_level
+--+-
-7407 | Haute-Savoie | 6
-8655 | Rhône-Alpes | 4
-74368 | Cusy | 8
(3 rows)
> From [1] I see that I am not the only one.
;-) Forgot all about this
> Has there been any progress
> made with this?
About what ?
The first step might be to feed the osm db : without boundaries, we've got
nothing. (see type=boundary or type=multipolygon)
In that part, I stumbed into the terrible truth : country boundaries are big,
france as a full relation export (with all ways and nodes) is ~30Mo osm data,
containing over 1800 members (thanks to our howfully numerous admin_level=8
admin division)
Still searching for solutions in the form of super-relation or else...
For the second step, I saw Marcus linked in the page you mentionned to country
specific speed limit and access restrictions. From there I think we can
either convert that in machine readable form and either tell all router
planners to make the good choice in absence of evidence OR construct a post
osm data feeder that will make those country specific things explicit.
--
sly
Sylvain Letuffe li...@letuffe.org
qui suis-je : http://slyserv.dyndns.org
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Which SRID in PostGIS for Mapnik rendering is needed?
('BOX3D(1492891.59943
> 6889040.048258077,1493197.347550918 6889345.796371222)'::box3d,900913)
> ERROR: column "addr:housenumber" does not exist at character 30
> STATEMENT: select asbinary(way) as geom,"addr:housenumber" from
> planet2_point where way && setSRID('BOX3D(1492891.59943
> 6889040.048258077,1493197.347550918 6889345.796371222)'::box3d,900913)
> ERROR: column "barrier" does not exist at character 68
> STATEMENT: select asbinary(way) as geom,"man_made","natural" from
> (select
way,barrier,highway,landuse,"natural",man_made,waterway,name,ref,char_length(ref)
> as length from planet2_line where waterway IS NULL and leisure IS NULL
> and landuse IS NULL) as roads where way &&
> setSRID('BOX3D(1492891.599437775 687.174201505,1493197.347550921
> 6889192.922314651)'::box3d,900913)
> ERROR: column "barrier" does not exist at character 78
> STATEMENT: select asbinary(way) as
> geom,"barrier","man_made","natural" from (select
>
way,barrier,highway,landuse,"natural",man_made,waterway,name,ref,char_length(ref)
> as length from planet2_line where waterway IS NULL and leisure IS NULL
> and landuse IS NULL) as roads where way &&
> setSRID('BOX3D(1492891.599437775 687.174201505,1493197.347550921
> 6889192.922314651)'::box3d,900913)
> ERROR: column "construction" does not exist at character 59
> STATEMENT: select asbinary(way) as
>
geom,"aeroway","bicycle","bridge","construction","foot","highway","horse","railway","route","tunnel"
> from (select * from planet2_line order by z_order) as roads where way
> && setSRID('BOX3D(1492891.599437775
> 687.174201505,1493197.347550921 6889192.922314651)'::box3d,900913)
>
> On Thu, Mar 19, 2009 at 7:02 PM, sly (sylvain letuffe)
> wrote:
> >> My osm tables use Google Mercartor (SRID= 900913) What is mapnik
> >> expecting? 3395 ?
> >
> > No it's good. If you've used osm2pgsql to populate the db, and if you use
a
> > copy of the osm.xml
> > ( starting with :
> >
> > )
> >
> > Then everything should be fine.
> >
> > Access restrictions at postgres level ?
> > Turn on the logs of postgres and try some custom basic queries to see if
the
> > data is here.
> >
> > select * from planet_osm_roads limit 1;
> >
> >
> > --
> > sly
> > Sylvain Letuffe li...@letuffe.org
> > qui suis-je : http://slyserv.dyndns.org
> >
> >
> >
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > http://lists.openstreetmap.org/listinfo/dev
> >
>
>
>
> --
> Ivo Brodien
>
--
sly
Sylvain Letuffe li...@letuffe.org
qui suis-je : http://slyserv.dyndns.org
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Which SRID in PostGIS for Mapnik rendering is needed?
> My osm tables use Google Mercartor (SRID= 900913) What is mapnik > expecting? 3395 ? No it's good. If you've used osm2pgsql to populate the db, and if you use a copy of the osm.xml ( starting with : ) Then everything should be fine. Access restrictions at postgres level ? Turn on the logs of postgres and try some custom basic queries to see if the data is here. select * from planet_osm_roads limit 1; -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql + relations of relations (aka "super-relations")
Hi, Relations becomes more and more popular (and complex). However in some cases (Europeen E roads), country boundaries, (huge forest ?) the number of members in some relation is huge. Motorway E 70 comes to mind with ~1700 members http://www.openstreetmap.org/browse/relation/27057 France boundary with ~1800 members Manipulation on those with the API is painfull. So we made a try in france with a super-relation : http://www.openstreetmap.org/browse/relation/11980 It's now much easier to edit but I don't know any tool (beside the home made parsers we've created) to display, convert or use it. Has anyone started work to support them in osm2pgsql ? or in any other tool ? PS: maybe other solutions for the database could be better ? -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to get offline maps of a large area in reasonable time?
> I started trying the first options but it is way to slow. Probably > because some never requested tiles also have to be rendered on the > server first? Exact. I don't think it's a good idea anyway without asking the server admin. That will put a constant load on the server and prevent other users to have a responsive answer to their browsing. > I already have the planet file > loadet into a PostGIS-DB > Does anybody know how many GB the whole world is? Or Germany? Doesn't your Operating System support "display disk usage" of a directory ? Maybe I don't understand your question... I'm working on the europe extract and I count around 3 times the uncompressed osm file or 30 times the compressed one (using osm2pgsql in slim mode) but dispending of your method it may vary very much. Or you mean tiles on disk usage ? > How long would it take to render on a quite powerfull desktop-machine? 3 seconds for zoom 0 3 times the age of the universe at zoom 30 That all dispend on the max zoom you want. As a rapid guess on germany at max-zoom 16 I bet it would take around 2 days > Or are there any other ideas on how to get such a big amount of data? See why almost no-one renders in advance every thing up to zoom 18 by reading : http://wiki.openstreetmap.org/wiki/Mod_tile http://wiki.openstreetmap.org/wiki/Slippy_Map -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API error 500 on /relation/*/full
> Either from wget or from JOSM those requests now only get a 500 interval > server error. Thanks to whoever resolve the issue, it restart working yesterday afternoon. It might or might not be a "who", maybe just the end of a load congestion. -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API error 500 on /relation/*/full
> Calm down... I'm not woried ;-) I am just checking the cause > A 500 error nearly always just means the object is corrupt > in some way. M Then it sounds an hurricane just pass over the database ;-) Those are almost randomly chosen relations from medium (~70 members) to huge size (~1700 members) and of boundary type. (+one europeen motorway E 70) $ wget http://www.openstreetmap.org/api/0.5/relation/8654/full $ wget http://www.openstreetmap.org/api/0.5/relation/8643/full $ wget http://www.openstreetmap.org/api/0.5/relation/11980/full $ wget http://www.openstreetmap.org/api/0.5/relation/44874/full $ wget http://www.openstreetmap.org/api/0.5/relation/7466/full $ wget http://www.openstreetmap.org/api/0.5/relation/8656/full $ wget http://www.openstreetmap.org/api/0.5/relation/27057/full $ wget http://www.openstreetmap.org/api/0.5/relation/7434/full None does download correctly. -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] API error 500 on /relation/*/full
Hi, I used to download relation from small to big size with a call like : $ wget http://www.openstreetmap.org/api/0.5/relation/8654/full Either from wget or from JOSM those requests now only get a 500 interval server error. Size might be the cause because it still works with very small relations. have we reached loads limits on the server ? did someone took ultimate measures ? -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [OSM-talk] Mapnik boundaries
On Friday 20 February 2009 14:09, Ben Laenen wrote:
>
> Since no-one seems to bother, it's likely these lines in osm2pgsql that
> are causing a problem here:
>
> else if( strcmp( type, "boundary" ) == 0 )
> {
> make_polygon = 1;
> }
>
> (in
>
http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/output-pgsql.c)
I remember being at the begging of this, I needed it for boundary conversion
to polygon, but just wanted a quick hack to do it. Don't really remember why,
but someone pushed it to the svn :
http://lists.openstreetmap.org/pipermail/dev/2009-February/013971.html
dev might well be a better place to discuss this, that code could be removed
if I'm the only one to use it (or pushed into a style's switch)
--
sly
Sylvain Letuffe li...@letuffe.org
qui suis-je : http://slyserv.dyndns.org
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Wiki support for compressed svg and/or any type of file
Hi, I have processed images of various size and type and thought the wiki would be a good place to host. But maybe it is not, or maybe there are other solutions. Here it is : http://wiki.openstreetmap.org/wiki/FR:Fond_de_carte_libre_de_france svgz format is not accepted by the wiki, and svg files I have are over 2Mo. I could down-scale the quality... but... that's "dommage" -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql support for relation type=boundary
hi, > On Tue, 2009-02-10 at 18:12 +, Thomas Wood wrote: > > I have no knowledge of c, osm2pgsql, geos or any of the other stuff, > > but I did have a hack at it a while back, and a very simple patch > > seemed to give me a usable result from my experimentations with the > > London borough relations. On Tuesday 10 February 2009 20:00, Jon Burgess wrote: > That will probably do exactly what you want. exactly what I needed. Many thanks. PS: for some reason I don't know, that's what I tried in first place but failed probably because of the -s switch or a little to old osm2pgsql code. -- sly Sylvain Letuffe li...@letuffe.org qui suis-je : http://slyserv.dyndns.org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
