Re: [OSM-dev] Building a friendly new editor in JavaScript
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 14-07-12 00:27, Paweł Paprota wrote: I wonder if for such a basic/welcoming editor it would be good to consider performance as a factor. Solution from Google Map Maker looks good at a glance (I have not used it too much) - you have to select a small area to work with so at any given time the number of objects shown does not degrade responsiveness. I rather fancy a theme based transaction. The theme defines what objects should be rendered to work on the specific theme. This prevents the accidental alteration of stuff that is unrelated, and gives a better focus on the task. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAlAAqIMACgkQYH1+F2Rqwn2BfACfXqUAKNtO1k2uxktd7sv3JiZU HBoAn357HIjLUUV7zrs/sx+fFj8ZV1DV =ncxP -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Suspend Imports / Bulk edits / Bots
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 11-07-12 17:10, Grant Slater wrote: Summary: Please stop Imports, Automated Edits, Bulk edits Bots until the redaction process has ended. What is your ETA? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk/9nB0ACgkQYH1+F2Rqwn3FPACfbjcLUeNVa0iCOY6RcwbL5bpW lz8An1RQWW0MNUwdUdmGrojm6pJEtO6i =vCti -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10-07-12 21:18, Even Rouault wrote: Following the recent brainstorming with Jukka, I've pushed into trunk a driver to read OpenStreetMap .osm / .pbf files . Is this driver able to replace tools such as osm2pgsql? Hence does it integrate node/ways/relations to points, linestrings, polylines and multipolygons? If it does so how does it know this information without a style file? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk/8gL4ACgkQYH1+F2Rqwn2b8QCfSTRy24WwGiljV1q2ZQAAiNrj IY0An3Y4mdCYaz1vftQj/9jD/FgGGr97 =9lxt -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Retina tiles - best way to support them?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 27-06-12 23:12, Robert Joop wrote: Why unreadable? What viewport setting do you use? A mapquest widget is used within a native app. angles I didn't saw you comment on was: going svg.gz all the way. Rendering tiles in SVG, doing so with a clue, for example fixed poit coordinates allows a decent compression over an easy zoom. How good is the SVG support on mobile devices? http://en.wikipedia.org/wiki/Scalable_Vector_Graphics#Mobile_profiles What about devices without decent SVG support? Serve legacy PNG available at fixed resolutions. Do you want to have two rendering pipelines, one SVG for higher pixel ratios and one PNG for less capable devices/browsers? Yes, don't break what ain't broken, and introduce a new better way that is slowly adopted. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk/sj9YACgkQYH1+F2Rqwn1g4gCghwBGrr/F843P8LO2gqOxaSqp vh8AnAk9t8VpC4pWuZ3vzaRemXuTCYWG =M6if -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Retina tiles - best way to support them?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 26-06-12 23:05, Frederik Ramm wrote: I have had some people asking whether I could supply them with Retina map tiles. Retina is an Apple brand name for higher resolution displays, and these users tend to mean tiles with twice the resolution. I am glad that Retina raises the issue, that is already present on Android, high resolution displays make the map unreadible. One of the angles I didn't saw you comment on was: going svg.gz all the way. Rendering tiles in SVG, doing so with a clue, for example fixed poit coordinates allows a decent compression over an easy zoom. The higher the DPI the more bandwidth is required to serve the tiles anyway, there must be some threshold where it is really more worthwile to render in whatever GL standard on the client, than sending over bitmaps as (effectively as textures). Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk/rP/wACgkQYH1+F2Rqwn3e0ACfZiFsvNudHwzFFPkvxnLbHi2s qPcAn3mviG0dSB6mqH5r6W5e5KVRP5h0 =f0jy -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Archive.org Tile Serving
Hi, Last night Archive.org has set up a collection to facilitate tile serving from Archive.org. We can experiment on what the best structure of this would be. Anyone with interest in this subject please contact me. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] converting mod-tile dir to flat files?
On 15-04-12 19:27, Sven Geggus wrote: What is the magic with this? How can I convert something like 11/0/0/66/61/112.png The Dutch tile server does do metatiling for rendering, but writes it out as static png files, served out with Cherokee as static webserver. It is a minor change to renderd. If you are looking for that, we have patches. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] converting mod-tile dir to flat files?
On 15-04-12 22:58, Sven Geggus wrote: Basically I'm looking for a script which will convert my mod-tile directory to something which can be served by a static Webserver. So nothing 'on the fly'? Only postprocessed? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem compiling mod_tile
On 20-03-12 22:04, elbereth wrote: I followed the mod_tile tutorial in the OpenStreetMap Wiki (http://wiki.openstreetmap.org/wiki/HowTo_mod_tile). My server runs Debian 6 with g++, I got the following error, while trying to compile mod_tile. I don't see your error, but if you are running Apache 2.4 I have a patch for that :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: Upcoming Emergency Maintenance / Downtime
On 13-03-12 22:14, Peter Körner wrote: Am 13.03.2012 16:24, schrieb Grant Slater: Sincerely Grant Slater On behalf of the OpenStreetMap sysadmin team Thank you very much for keeping things running! You're the ones, that keeps our great project running and running and running and running... At this time you do wonder... one faulty motherbord... why not a nice slave server. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: Upcoming Emergency Maintenance / Downtime
On 14-03-12 01:43, Grant Slater wrote: We have a machine ready to be a slave, but are not yet ready to enable it. We are planning this for mid/late April timeframe. Very good to hear :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Automatic modifications to OSM data
On 12-03-12 22:14, Chris Hill wrote: On 12/03/12 20:40, Morten Olsen lysgaard wrote: I'm running a sister project of OSM, OpenAviationMap. ^^ My first reaction is that airspace does not belong in OSM. Chris, please read what he actually writes. But now we want to update the data that we've already uploaded. Does anyone have experience with this in the OSM community? In The Netherlands there have been two occasions where such thing happened. The initial AND import an the 3DShapes import. In both cases were areas that were really high quality marked and excluded from an import. Maybe you could do the same? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 03-01-12 09:21, Philipp Borgers schreef: We can provide rack space and bandwidth for at least one server here in Berlin. We also have some servers we can contribute to the project but they are not that powerful. I hope you can put your offer on the Wiki; http://wiki.openstreetmap.org/w/index.php?title=TileCDN A real real CDN just requires local bandwidth, a harddrive and a static webserver. It is therefore interesting to offer 'local' end nodes there were traffic is free due to peering, for example. The major problem are low zoom level tiles which are rendered adhoc. We can't solve this problem with tile caches as far as I know. Should every node in the CDN be able to render tiles? No, the CDN nodes will al be static. Some intelligent behaviour is envisioned for tiles that are not rendered. Is there some kind of global render cluster where caching nodes connect to? The other way around, nodes receive updates, push based. How do we keep the data/database for rendering in sync? Like every other tile server does. And maintaining central information on last updates per participant in the CDN. Do we offer different tile styles? First things first: the default map style. Is CDN usage free of charge? Yes, obviously. I envision that something can be done using the attribution text to present a status/powered by notification. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk8DRcwACgkQYH1+F2Rqwn1gCgCgi7xMqCuY3xbvXTWjzEwKamHn LAAAnjZpAw0qFtQUsuuNKu9pRZ0Ql+mz =j+uU -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Archive.org replied positively; update on the wiki. http://wiki.openstreetmap.org/wiki/TileCDN Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk8CLMQACgkQYH1+F2Rqwn0PkwCggIDTSQtuqNYVUFo/8HoQsbgA dvIAni4EbJnciIOmiKRcw3qfiklWjX8f =ubKr -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 The basic outline; - - People seem to love using tile.openstreetmap.org for any of their apps - - Apps get blocked for various reasons, contributing back is difficult - - OpenStreetMap wants to be the leading provider of such data I would propose a small working group to outline what should happen to facilitate the creation of a Content Delivery Network (CDN) serving the basic tiles, and allow easy contribution this network. This should produce a prototype of at least 3 servers. Questions such as: quality, update frequency, traffic shaping, geographical balancing and high availability could be part of this working group. I would like to invite anyone to participate, especially: - people that already have their own tileservers running, and/or are currently balancing traffic; - business folks: what could be a motivation and what can be a cutback in for example attribution, - users of for example openlayers, etc. what kind of caching can be applied, and if this should be configurable client side Participation can announced in private or on list. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk798H0ACgkQYH1+F2Rqwn2I9ACeJV+vFfBvI4CaTET0BqjuuX1s G78AnRv6+Qh7MkIrN1DQ6fQ34p1j0uuS =+WNS -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I have just summerized the e-mail by Mike, and created a wiki page the follow up the status: http://wiki.openstreetmap.org/wiki/TileCDN Op 30-12-11 18:23, Lynn W. Deffenbaugh (Mr) schreef: I would love to monitor such discussion and contribute if I think I have something to offer. Will such discussion be here on the [OSM-dev} or elsewhere? I don't want to induce a lot of offtopic and negatively. The 30 days in the subject is something that I would like to realise, not talk about. Theoretically the CoralCDN could be used for distributing OSM tiles by simply suffixing the domain with .nyud.net, but I have not tried this yet. There would be little to no intelligence brought to bear on how long individual tiles would be cached vs flushed and/or whether mixed-revision tiles would be present in and delivered from the cache, not to mention that I don't think the CoralCDN passes through the original User-Agent either. I do not have yet investigated what the options are of CoralCDN, but the main problem is to get the *alternative used*. I guess switching to a system that is partly in our own hands results in a higher quality, thus we must have ability to invalidate data quickly. If that is covered in CoralCDN the only thing remaining is finding a good solution for the 'hostname' issue. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk7+AE4ACgkQYH1+F2Rqwn3V1ACfd9wODnhcav/lC0jnaK816/AX YAwAnig+5IuGoiuxciB32FHl3ulnMvjk =pKVk -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Christmas present ....
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-12-11 12:45, Simon Poole schreef: PS: the server is a Hetzner EX-5 so please be nice to it and don't stress it too much :-) If this is really the ODBL only-map. Then I have some doubt about its correctness; OSM has some functional eastereggs too. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk713IAACgkQYH1+F2Rqwn3jlACbBozJOmupJ4ScQCwopW2hRf39 SnIAnRkOZDUQZtsVLVftU70gvrv1FKnh =if3K -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Christmas present ....
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-12-11 15:19, Simon Poole schreef: You did read the information displayed before you viewed the map? :-) Actually I did: hide objects that were created by mappers that have not agreed to the OSM contributor terms, exceptions below So that means that if I created an object, I'm in The Netherlands, it should be hidden right? Given that I still can't agree to the ODBL. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk714J8ACgkQYH1+F2Rqwn0VgACfdFjhdJ+QMP9/9OF4fAOMnC6S lw0An2uWJjOTSwvT403DeseKAr6AW2cL =cHZS -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] php port of OSM api
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 19-12-11 02:55, Tom Sparks schreef: Is their a php port of the OSM api? if not what are the bare minimum needed to support polatch? Yes, I did coded one once... Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk7un60ACgkQYH1+F2Rqwn28KwCcCpSc8k/ak/Tg0mMOJq0fjGam s+IAoI2UIx3WoGHnnRFlSjC2XJOmRy2R =BBtc -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] php port of OSM api
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 19-12-11 02:55, Tom Sparks schreef: Is their a php port of the OSM api? if not what are the bare minimum needed to support polatch? Ah wait... you want a server, not client. I only wrote 0.5 in plain C... Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk7un9EACgkQYH1+F2Rqwn3uxwCgj6pI4G0MMsTXT9p38hRMUkGU IEAAoJCRHsWU8O9VZz2plLrrbXPQ7MVo =FkkW -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] CORS headers on OSM tiles (for WebGL use)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 17-09-11 23:46, klo...@gmail.com schreef: Would it be possible to add the CORS HTTP header: Access-Control-Allow-Origin: * to the OSM tileserver when a tile is requested? Probably you will need the other two headers as well. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk511oEACgkQYH1+F2Rqwn1xawCfcpep+3hWvlZ41cbs+1Xd9eOQ n3oAn3MyzPFKJ9tC34Tfshcw0zITz5Qn =ORKS -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] CORS headers on OSM tiles (for WebGL use)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hey Steve, Op 18-09-11 20:27, Steve Coast schreef: What are the other headers? I thought it was just the one mentioned. It is browser dependant, firstly; - - OPTIONS request should be allowed (if that results in a 405/503 = fubar) Then the headers; - - Access-Control-Allow-Origin: * ; Takes care of the 'real cross domain stuff' - - Access-Control-Allow-Headers: X-Requested-With,Content-Type ; Both seem to be needed 'sometimes', in some browsers. And to give an heads up: * is not accepted here. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk52PCwACgkQYH1+F2Rqwn12hACdE6J0wbSPaApOfwmL/dUPXe1t Jw4An2JJYYxBGKY29rr8OeY8iGoy39Ia =dpvZ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] CORS headers on OSM tiles (for WebGL use)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 18-09-11 21:02, andrzej zaborowski schreef: I can try preparing a mockup of osm.org with that option if there's interest. If I enable this on tile.openstreetmap.nl, you could try it out for The Netherlands ;) Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk52QYUACgkQYH1+F2Rqwn1bTwCfaJxUGUR36Qy8oOq4ajF08fHz CrsAniZn4976Q6Cu9DE2zUoIe0oE2Vye =dQ+C -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Proposal for OpenMetaMap - proper OSM import solution
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Peter, Op 18-08-11 12:11, Peter Körner schreef: Will it be in the planet.osm? If not, how are tile-providers are supposed to draw an icons for museums? Like they do now for coastlines... there are external datasources and they are already in use. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk5NGO8ACgkQYH1+F2Rqwn2eKQCdEfJSNJ19m3X80drEk/JHDJOn nGoAn0im5FMhd2/dSNXXzDyzFGky5KAz =IXAg -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Proposal for OpenMetaMap - proper OSM import solution
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 15-08-11 22:57, Jaak Laineste schreef: My proposal is in http://wiki.openstreetmap.org/wiki/OpenMetaMap , any positive and negative comments (especially help to actually implement it) is most welcome. I think Google translate shows you are not alone with this idea; http://blog.openstreetmap.nl/index.php/2011/05/26/durven-denken-over-een-gelaagd-osm-model/ I think it is interesting to think about editing objects from source in OpenMetaMap, 'applying a stylesheet' so a source can be rendered using a 'standard' stylesheet (thus: complementary key/value pairs, not changes in Geometry). You have also been thinking about rendering. I think it would be awesome to firstly try to get WFS interoperability. If any WFS is supported in the rendering chain, based on the OpenMetaMap 'translations'. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk5Jj+MACgkQYH1+F2Rqwn0T3QCfWeL93vNXLNyttWLkO+UJN02F nvUAn1/W3CXQXTsq6qa6JJtYWdCA+hKh =mn58 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] (mod_tile) moving png files to a /z/x/y
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 22-05-11 17:12, Philippe Coent schreef: I am running a mod_tile/renderd/postgis map OSM server. I rendered all the meta tiles using the render_list tool. I can then convert those .meta files into .png using the provided tool. On the Dutch server we have basically a 'modified' renderd, that outputs non-metatile output, but renders in metatiles. Would that be interesting for you in the future? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk3ZVqoACgkQYH1+F2Rqwn2aXQCcCwnWJQdguJTTIdFeWroPDk5i CwMAn3LDGoPMZeHYHZ4PE8LKP8oviSzL =Ifci -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 21-05-11 12:52, Frederik Ramm schreef: On 05/21/2011 03:53 AM, Igor Brejc wrote: Can you give some rough estimates on how much time we still have until this 64bit issue comes up? My rough estimate would be that it *may* happen around the end of this year, but more likely around mid-2012. Of course this depends heavily on how good we are at stopping data importers going crazy ;-) Just look at http://wiki.openstreetmap.org/wiki/File:Osmdbstats2.png and ask yourself when that line reaches ~2 billion (take into account that the line only indicates the number of visible nodes; the highest node id tends to be about 20% higher than the number of visible nodes). Given the upcomming 'license change phase 4' wouldn't it actually be a good idea to 'start over' with counting again. Ditch the complete history and reimport everything? This might sound 'rücksichtlos', but does the memory and storage overhead introduced (now) really benefit the user? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk3XoCMACgkQYH1+F2Rqwn0VMQCggJNdFMOFHwxuWvd8KXfARU6n nEkAmwT5eSoxWrfwAFEP+ML3WxNeXjy7 =BpoK -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 21-05-11 16:52, Jochen Topf schreef: If we use unsigned ints we have some more time. Problematic would only be a few cases where negative IDs are currently used (like in JOSM for data thats not yet uploaded to the server). But it seems wasteful to me, to go to 64bit a year or so earlier than needed to accommodate this case. Don't forget the binary format (PBF) also was basically twisted around to accommodate for negative numbers. Your suggestion makes sense, there shouldn't be negative IDs. And additional making the value NOT NULL in the database, also gives one bit extra. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk3X174ACgkQYH1+F2Rqwn2pfACfRSmILv18t39YecJjrJxvOTaN o7sAmwb5xMnHPwkKYtAGu1fcSGydg1AE =H/3R -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [GSoC] Video based speed limit detector
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 22-03-11 08:33, Kai Krueger schreef: I'll possibly be able to mentor such a project, although I know little about the code of any of the editors, so I'd be less able to help on that side of things. Since I was the mentor of the last project, there is a great number of test material available to even build a recognizer. Video segmentation is step two, not the first step. If someone isn't able to even find a sign on a still image, it is even harder to do it on motion pictures. For Dutch signs, and most likely many international ones on Wikipedia SVG images do exist showing signs in the highest details possible. So first things first: - sign is present (x,y,w,h) - classify sign - segment video - enhance recognitionrate on multiple images - pinpoint the location of the sign in 3D Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk2I6C8ACgkQYH1+F2Rqwn3u/wCggw+qJzPbUuR60IzOclFlz3f8 i7gAnRm2+D2cBPWT3+Fd2hKIKdKghJqS =reZv -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
On Mon, 10 Jan 2011, Matt Amos wrote: it seems postgres 9 isn't able to do temporary table creation on hot-standby servers [1]. maybe this is something that'll be fixed in future versions (but it doesn't seem to be on the TODO [2]) or maybe it's an opportunity to put a bounty to good use. I've currently been experimenting by application level replication by using XMPP. Technically any subscribed client could get 'old' messages when reconnecting. Since it is trivial to get a Python XMPP client running, it is pretty portable, with respect to databases and platforms. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Delta-encoding in PBF
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
Op 18-12-10 11:53, David Paleino schreef:
Hello people,
I'm writing a parser in C# for PBF dumps. However, I'm having issues with the
Delta-{en|de}coding.
http://git.openstreetmap.nl/index.cgi/pbf2osm.git/tree/src/main.c#n588
Maybe that helps?
Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEAREKAAYFAk0MqNkACgkQYH1+F2Rqwn08DgCdFg0+MpUVGwkyRVMTPhqOlWFR
CjIAniL8AqWbBMhU1LkwQWT0qMZOPuBW
=kyDH
-END PGP SIGNATURE-
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Script to extract bbox from history planet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 16-12-10 09:22, Frederik Ramm schreef: If someone feels like (a) re-implementing this in C I think Mitja did this in the GSoC program this summer (the cutout thing). It would be valuable to figure out a way to do 'proper' full ways and relations in a low memory setting, but probably I/O itself is always an issue too. I can imagine that you create a cutout of the nodes first, quicksort it in a mmap-file and then run it over the ways table. Storing a mmap file with only the way ids,quicksort on it and walk over the relations part. Now a second run would grab the missing ways, and the remaining nodes. I wonder if it can be done more efficiently. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk0J2EYACgkQYH1+F2Rqwn2GZACdGY9IsHcY0lNBhymukjztL62P 9xoAoII1oyPOfGd1bjpaGTBPu8oihJIt =80mv -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API/XAPI caching proxy server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 16-12-10 22:10, Peter Körner schreef: The osm2pgsql scheme does not store meta information (version, creator, timestamp), which are required to create valid .osm files. Valid sounds like someone updated a XSD file already? http://wiki.openstreetmap.org/wiki/API_v0.6/XSD No mention of version, creator, timestamp as required :) Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk0KglgACgkQYH1+F2Rqwn1Z1wCdHCQfSxyMsAShl0XPBVEH4+7p HS8AoIY544/hqVix0vBRRQO4+/P4Q0p9 =EJoD -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API/XAPI caching proxy server
On Wed, 15 Dec 2010, Frederik Ramm wrote: On 12/15/10 08:53, Stefan de Konink wrote: It has been done in C already :) By different people, so what are we waiting for :) If there's a working instance that supports the XAPI protocol it should be added to http://wiki.openstreetmap.org/wiki/XAPI#Servers and also added to the automatic load balancing. Sadly we cannot defined partitions there yet. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API/XAPI caching proxy server
On Wed, 15 Dec 2010, Frederik Ramm wrote: But then again: XAPI has been written by a single person, in (what we consider) an esoteric programming language, in their spare time. So if someone really wanted to do it better, it should be a breeze to implement the same, or even an improved, interface with a standard toolbox. It has been done in C already :) By different people, so what are we waiting for :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API/XAPI caching proxy server
On Sat, 11 Dec 2010, Wyo wrote: To have as many as possible proxies running in the internet means to stick with technologies easily available. That limits the choice almost entirely to a PHP/MySQL solution, at least for the beginning. DBSlayer is actually what you want here. Discussed before on this mailinglist, sadly at that time nobody wanted to do distributed databasing. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM2PBF C-development, wanna participate tonight?
Hi all, I would like to invite anyone that would like to volunteer in osm2pbf c-development to join us for a tonight coding session. I suggest #osm2pbf We will build the basic building blocks, analyse documentation, and hopefully compair output to osmosis, in speed and what it did produce. For starters we have fast parser, so we can completely focus on compliance. Some stuff that has to be investigated is xz (lzma); so it can be included in pbf2osm as well. If C is not your language of preference, but you can read Scott's documentation and/or the Osmosis Java implementation, documenting the format can be a good task as well. I'm thinking about a nice DFD or Flowchart. You can reply in private, or just come by tonight. I guess around 19:00 CET, would be a great moment to 'really' start, I'll try to have as much ready before. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM2PBF C-development, wanna participate tonight?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Mike, Op 11-12-10 16:26, Mike Dupont schreef: I have done some work on an osm parser in c https://code.launchpad.net/~jamesmikedupont/+junk/EPANatReg and pbf reader in c, https://github.com/h4ck3rm1k3/OSM-Osmosis/ you might find it usable. i am still working on other things, but will be willing to review code. Thanks for the follow up Mike. Was currently thinking to reuse, if better (or: faster) methods are available. Open for suggestions. http://repo.or.cz/w/handlerosm.git/blob/d2711828e5aeb38dbc4c8d870ec11967da51833f:/osmsucker-ywk.c and the pbf2osm code. http://git.openstreetmap.nl/index.cgi/pbf2osm.git/tree/src/main.c?id=35116112eb0066c7729a963b292faa608ddc8ad7 Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk0DmgIACgkQYH1+F2Rqwn09NACgk4qr4yuHcbJHNJrdQTOaoR9j 0McAoItBB48vBvIWIcnsTna/WbL7y3n3 =tYfW -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM2PBF C-development, wanna participate tonight?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 11-12-10 22:05, Anthony schreef: On Sat, Dec 11, 2010 at 9:58 AM, Stefan de Konink ste...@konink.de wrote: I would like to invite anyone that would like to volunteer in osm2pbf c-development to join us for a tonight coding session. I suggest #osm2pbf I hope this isn't a stupid question, but... What network? You have found it already, for others: like OSM: oftc. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk0D66kACgkQYH1+F2Rqwn18rQCfXMYFlZwVVFhowSQHXchWTeGr nU8An3gyQjbwCTgdp7etgxDgd2brJy/o =Km0A -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HALP! transiki needs YOU!
On Wed, 8 Dec 2010, SteveC wrote: Transiki is like OSM but stores magical transit data. It's approximately, nearly, at 0.1. It looks and feels very similar to OSM internally. It's written in rails, it's API looks similar and so on. You can think of it as a baby OSM. I'm a little overwhelmed and haven't had the time to review patches sent in, update the site etc. So, if you are a rails person or want to be and have a couple of hours please have a look at transiki. Anyone contributing will be sent an I Love You bean. If it is like OSM then I think you are focussing on static data. Hence it would be as non-innovative as Google Transit. For a Dutch variant OpenOV.nl (Open Public Transport) we go beyond static data, we go for realtime positioning and sharing this information using open standards and open protocols. This data is available as well for many governments, transit providers, etc.. And best of all, we are only a HUB to the data. So I hope people realise that they can request the same data as Google does. And going realtime is where you want to be in 2011. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HALP! transiki needs YOU!
On Wed, 8 Dec 2010, Mike Dupont wrote: realtime could be provided by such a thing as GEORSS http://www.georss.org/Main_Page where you show either, where the things SHOULD BE, or another feed with where they have last been reported to have been. Our design documents include GeoRSS feeds over XMPP. Currently implemented with some python code and ejabberd. So a 3rd party subscribes to an active route, gets data, unsubscribes. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HALP! transiki needs YOU!
On Wed, 8 Dec 2010, Mike Dupont wrote: Ok Stefan, I am sorry that I have not read about all these things you guys have been doing. please tell me, how do you get all this data? Is there a way to extract all the transit data from osm in an easy way? that would be a good test for transiki to import this. Most of our data goes into OpenStreetMap and does not come from OpenStreetMap. OpenStreetMap is primary used for geo-semantical stuff recovery. In The Netherlands, by law, any transit provider has to offer their data to any national transit routing provider. Inventive as we are the government even defined a standard for that called 'BISON' they are focussing now on the european variant. In every respect both are bloated and the Dutch one is plain ugly. But like OSM, it works. Level 1 is basically the static and positional data. The aditional levels include updates and realtime information exchange. All this was done prior to the deployment of the Dutch National Databank for Public Transport data. Our insentive is basically to help define, and implement a reference infrastructure for this 'hub'. This reference might become the actual thing. Train information is a sperate issue, currently there is a realtime system that manages that located at the governmental company that manages the complete traininfrastructure (opposed to the companies that drive the trains on them). The idea is to have their system as client to ours, and inject the data that can be subscribed to. Because our system is primary targeted to The Netherlands and to data exchange, opposed to harvesting as much as you can eat, the documents are in dutch, but easy to translate. I wouldn't mind to send in a lot of Dutch content to your project. But please understand that timetables is something that is in the process of being phased out anyway... and routes are basically 'trivial' to get. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HALP! transiki needs YOU!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 08-12-10 21:30, Mike Dupont schreef: On Wed, Dec 8, 2010 at 9:22 PM, Stefan de Konink ste...@konink.de wrote: But please understand that timetables is something that is in the process of being phased out anyway... and routes are basically 'trivial' to get. Ok, well if you have data we can try and import into transiki db, I will look into it. so you have the position of the trains? Positions of trains is currently being worked on (basically getting the specifications etc.) There is some preliminary support for positions of some busses for one bus company that has this data 'sometimes' available. For about 50% of all Dutch busses the data is currently being collected per minute for management and Dynamic Travel Information Publishing. In principle 2011 would allow to connect to that 'huge' mass. What we made available on the Dutch OSM mirror is busstops, and 'known' bus routes. But some routes still need to be integrated. Anyway I hope a suite datamodel is defined, topology based. Also we need to look at generating wikipedia articles on the routes like the have for the paris and ny metros. Sounds like a job for XSLT ;) Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz/7bEACgkQYH1+F2Rqwn1q7gCfd9CDTKhQ+PRY6TNuWnloHcei C/wAnRUwilQQZZbchmydbRdB5m98smMz =Jvcs -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What means 500 Internal Server Error on API server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 08-12-10 22:13, Wyo schreef: And how do I bring an aquivalent error message onto the screen when this happens? You don't because you only know that someone went wrong. But not what actually did. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz/9WAACgkQYH1+F2Rqwn1LBwCcDP6Jqt2pa45Th7e02U7NE7mL N+8An32faMk0Q8Gj72IeqUV5ZK0pKxzB =vXKD -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Wed, 1 Dec 2010, Anthony wrote: On Tue, Nov 30, 2010 at 11:03 PM, Anthony o...@inbox.org wrote: On Tue, Nov 30, 2010 at 8:29 PM, Stefan de Konink ste...@konink.de wrote: If any of gzip/bzip2/lzma in the general give better compression ratio's (20% smaller), then this compression scheme should become the default format. Depends on the performance. If all you want is max compression without regard to performance, you're almost surely better off using raw and then compressing the entire file with LZMA (e.g. 7zip or xz). LZMA vs. zlib actually makes less of a difference than I thought it would: -rw-r--r-- 1 a a 103M 2010-12-01 08:07 florida.osm.bz2 -rw-r--r-- 1 a a 129M 2010-12-01 08:32 florida.osm.gz -rw-r--r-- 1 a a 74M 2010-12-01 08:19 florida.osm.pbf -rw-r--r-- 1 a a 169M 2010-12-01 08:15 florida.osm.rawpbf -rw-r--r-- 1 a a 62M 2010-12-01 08:15 florida.osm.rawpbf.xz -rw-r--r-- 1 a a 86M 2010-11-25 11:29 florida.osm.xz I suspect it would make *much more difference* when it comes to the full history .osm, though. Does PBF support full history files? Does Osmosis? Did you benchmark what pbf + lzma did or did you embed lzma in osmosis? Stefan___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Wed, 1 Dec 2010, Anthony wrote: Did you benchmark what pbf + lzma did or did you embed lzma in osmosis? xz uses lzma. I made an uncompressed pbf file (florida.osm.rawpbf) and then compressed it with xz (florida.osm.rawpbf.xz). This isn't the same as making a pbf file which uses lzma, but it should be a good approximation of the compression achievable by embedding lzma in the pbf. Yeah, but your lead basically shows we are talking about more than 10%... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Wed, 1 Dec 2010, Anthony wrote: Yeah, but your lead basically shows we are talking about more than 10%... Yeah, probably, but at the expense of more complicated code, greater memory usage, etc. The hole process is IO-bound... memory is used anyway to overcome the IO issues... I'm interested now in seeing how the full history compression goes, though. If it can achieve 70, 80, 90% on top of zlib, then it might be worth embedding the compression as opposed to just using it for transfer over the Internet. The dictionary is compressed per block, so it greatly depends if the trick works. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Wed, 1 Dec 2010, Anthony wrote: Not in an embedded system, which is where a small difference like 10% is going to matter. Please elaborate? Either the memory is used for a block cache or for the program. I'm interested now in seeing how the full history compression goes, though. If it can achieve 70, 80, 90% on top of zlib, then it might be worth embedding the compression as opposed to just using it for transfer over the Internet. The dictionary is compressed per block, so it greatly depends if the trick works. 32 megs is a lot better than 900K, though. 900K is how much zlib uses, right? I don't get your point here, what do you mean? Do you mean that the memory requirements for zlib is lower? Because don't forget that the extracted piece is kept in memory + the deserialised version. Which is basically much bigger right? Stefan___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 01-12-10 17:30, Anthony schreef: Anyway, I'm probably completely wrong about this. Sorry. I guess the fastest way to verify all this is someone that adds the LZMA and BZ2 library to java and check in osmosis. Your numbers give me the impression that it is worth to pursue different compression strategies. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz2gGcACgkQYH1+F2Rqwn1tLACfbuO+z3uLarrQ/BUUkkmHsfvX 2mIAoIlrEHucqWkmz6DV8z+9OkSDT3kf =5p2d -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Tue, 30 Nov 2010, Jochen Topf wrote: The PBF format supports three compression types: zlib, lzma, and bzip2. Do we have to support all of them? What is the currently existing software using? IMHO it would make more sense to just define one and stick with it. Easier to implement for everybody, less reliance on external libs. Why this mentality? It is trivial to implement a decompression algorithm and some work better than others. Sounds like complaining about stuff you don't have to care about. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Tue, 30 Nov 2010, Tim Teulings wrote: Hallo! Why this mentality? It is trivial to implement a decompression algorithm and some work better than others. Sounds like complaining about stuff you don't have to care about. I would not implement decompression myself, I have better things to do. Thus I would a library for this. A library however is a dependency, that must be build installed and delivered (not all the world runs Linux with smart packaging systems) and licences have to be checked. It makes sense for teh developer to try to reduce such dependencies by agreeing on one standard compression format. Since all program interfaces virtually equal gzip, and it gives perfect extendability. The choose for supporting the 3 most used compression schemes is perfectly sound. Stop wining about code that either you do not write, or didn't care about before. There was this great moment when the bitstream was defined, and absolutely nobody cared until people started to write pbf code. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, Op 30-11-10 20:49, Frederik Ramm schreef: Stefan de Konink wrote: This is the place for the 'too little, too late'. We are beyond the point of 'what' the bitstream should look like: you ought to handle what is defined now. This is not how we work in OSM. We don't have standards. For some reason we do, this is not a free form fight. And if we can change the API every week, I wonder why we are still at XML then. We can change stuff at any time, and indeed I would not hesitate for a second to change something in the PBF format if it turns out to repair a design problem or bring great benefit. (If it were my call which it isn't.) The only reason your friend/collegue Jochen started to ask about it is because he found it difficult to implement 4 ways to encode/decode the data, which are in principle the same. So what that your tool doesn't support a specific extension? If that compression is often used, who are you fooling? Are you suddenly caring about linking -lbz2? I really don't like your attitude. It's great that you took the time to write pbf2osm but it seems you expect to be revered for it. You give the impression of someone for whom coding something is only a means to climb onto a platform from where he can heap spite onto others. (I remember you derogatory comments about C++ while you wrote pbf2osm, and putting comments like osmosis devs failed to read the specs in one's code is not exactly a sign of maturity either.) Whats your point? I also wrote the entire API 0.5 (R/W) and XAPI in a C extention to a webserver. Ab-so-lu-te-ly nobody cares what I (or probably anyone else) writes here, it was interesting that after 2 weeks of publication Lennard came up with some detail that everyone who would have checked the output could have come up with after the first day the code was published here. My point is pretty clear, you want the threat PBF as something that is in flux, I observe that feedback was requested and (virtually) nobody cared. Protocolbuffers is something that can be extended. If someone would actually CARE baout removing certain compression techniques he would benchmark the compressionalgorithms on the data presented and not start in a: I do care that it seems I am writing code that might never be used. ...so all code of Jochen should be used now? Get real. So exactly what Scott suggest: why does nobody step in then, write code that nobody uses afterwards and present a proper benchmark to show that bzip/gzip/lzma is useless? Then you probably also noticed that it is still a (huge) open question to write a regression testsuite for all parsers and generators. And since the general opinion is now that nobody wants to move until there is a second implementation of osm2pbf (instead of actually switching), everyone is waiting this greatly annoys me and probably not only me but also the guy that actually took great effort to define the protocol and review code of others and answer questions. What exactly is your problem? PBF is alive and kicking. I'm using both Osmosis PBF support and your implementation of pbf2osm on a daily basis, and many downstream users of Geofabrik do the same. This is my problem: http://planet.openstreetmap.org/ And the fact that protocol buffers probably would make the API far more efficient. I find it totally respectless that *you* are now doubting his qualities but didn't step forward when feedback was asked. Excuse me, but discussing potential problems of a design is not a show of lack of respect - unless presented in a form like the aforementioned osmosis devs failed to read the specs. Oh dear, so because I actually feedbacked on Scott and asked questions, and verified my code and implemented the specs I cannot complain osmosis didn't? Sounds like we cannot bash IE6 anymore because it did an effort to implement HTML rendering... Why does this subject get me so angry? Because the request shows lazyness and not an effort te show that something is useless because the compression algorithm are not suited. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz1YXQACgkQYH1+F2Rqwn2+UwCglRWja5rs5jYs4iFp9C/PgJuE Vw8An01ZXFsY6XFcFhEDDC9NP4B705W6 =l28+ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Storing way/relation start offset in PBF file
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 30-11-10 21:26, Jochen Topf schreef: Sometimes it is useful to read ways before nodes or relations before way or nodes. With the XML format this is not really possible, but with the PBF format it could be reasonably easy if we store the offsets in the file where the way and relations start, respectively. If we write the offsets at the end of the file, we can still do streaming write. When reading from a stream you have to read everything anyway, when reading from a file, you can seek to the end and find out about the offsets and then seek there and start reading the data. Is this something we can fit in the existing extension mechanism? If not, its not a big deal, but we can note it down as possible extension in case we'll do a new version of the format someday. There is currently a potential indexing extendability. What you want would only work when everything is in fact nodes, ways, relations. But will not work if it is ordered by: bbox(nodes, ways, relations). I guess an index that specify the offset of each block is pretty useless, and an index is only useful if the file is fully structured in the 3 node, way, relation blocks. Most likely it would be much more interesting for most users to apply a spatial index (like what is currently being done with the Garmin maps) to the data to partition it. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz1a4MACgkQYH1+F2Rqwn0BPgCeIUqyQIozUhz4T//QN/bzt2sY RlAAn3IiTVOTyz7VYhMVicT5AgbI2fYW =m8bB -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Scott, Op 01-12-10 00:41, Scott Crosby schreef: The real question is does supporting bzip2/lzma offer advantages that are commensurate with the added implementation complexity, not just in pbf2osm but in every other reader too. If any of gzip/bzip2/lzma in the general give better compression ratio's (20% smaller), then this compression scheme should become the default format. Since (sadly) PBF goes into an 'archival' format opposed to a wire format. Would you be willing to run an experiment with LZMA? If it shaves a gigabyte off of the planet, then I'd say its worth further consideration; if it shaves 100MB, then its not. Make a case for why it should be included. I completely agree. But experimenting with LZMA means first a osm2pbf that supports LZMA. And currently I feel that the only 'true' tool that should do something like this should be named pgsql2pbf. I honestly cannot find a single reason why it would be good to use the XML as intermediate format, except for legacy support. Excuse me, but discussing potential problems of a design is not a show of lack of respect - unless presented in a form like the aforementioned osmosis devs failed to read the specs. Oh dear, so because I actually feedbacked on Scott and asked questions, and verified my code and implemented the specs I cannot complain osmosis didn't? You do realize that *I* designed the format AND wrote the spec AND wrote the osmosis reference implementation? No, I didn't. But my archive also states that James Michael DuPont also published his OSM-Osmosis version. indepth And for the reader; that only was presented my flame to the osmosis implementation; Out of the blue the OSMOSIS implementation started to introduce -1 userid's, this is in no place documented, neither is it a default at present to represent past anonymous edits with a negative userid. Especially since at that time the uid's couldn't be negative (by spec) and the format specifies 'has_uid'. /indepth That means that if there are any errors or omissions in that implementation or spec, they are my mistakes. If there is an ambiguity, then I have made the call as to what is right. If there are any differences between the spec, reference implementation, and the conceptual design, I'm the one resolving the conflict and determining the best way to fix the issue. Since the current osmformat.proto still has a int32 for a uid, which is in fact always positive number in the openstreetmap database, the problem has been reported before. Would be obvious to haven't defined it at all in message Info and use 0 in DenseInfo. I do appreciate you finding the bugs and ambiguities in the spec by being the first independent implementation, and I hope you will consider running the LZMA experiment, but you have been rude and insulting. Basically you are asking me to run tests that Jochen should have come up with to prove that your specification of multiple compression formats sucked. I find this insulting. I think your choice is sound, and if a tool doesn't implement compression scheme X, then just inform the user. And if you found my comment in the code rude and/or insulting, I would have expected an email of you in private about two months ago, because honestly something by-far more rude was written there. But again, nobody seems to care what happens here or what is written. It is not strange that a flamewar over a format starts seven months after initial publication or that a pointing fingers at code starts about two months after the publication of it. I do find it interesting someone actually bothered to read the code, sadly I cannot speak about any broad collaboration. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkz1pP4ACgkQYH1+F2Rqwn2JHQCbBbYJN0EiYFCgtF2bQCP+CsVm MA8AnjrA8bV/Tk8JE9KnqB78xwm6ma+b =X7fJ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Our beloved XML format doesn't parse; planet 11-11
On Tue, 16 Nov 2010, Jon Burgess wrote: Are the errors you see consistent between runs? Basically 'still' importing now. I can possitively confirm that this is because of 'out of memory' situations. You may wonder if instead of using memory osm2pgsql shouldn't go for a mmaped areas (so instant swap). I guess either some operations are not checked for null pointers, or something else fishy is going on. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Our beloved XML format doesn't parse; planet 11-11
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, After 4 unsuccesfull in osm2pgsql runs, and today after md5 validation. I am pretty sure that the file that I have downloaded is complete. ea628ba9912e6d33ef65d35b7cbe6823 planet-latest.osm.bz2 But osm2pgsql reports: Reading in file: - Processing: Node(832188k) Way(7341k) Relation(0k)Entity: line 1275668752: parser error : Specification mandate value for attribute v tag k=source v ^ Entity: line 1275668752: parser error : attributes construct error tag k=source v ^ Entity: line 1275668752: parser error : Couldn't find end of Start Tag tag tag k=source v ^ - - : failed to parse So dear XML lovers, what went wrong with with the planet generation? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkzhpekACgkQYH1+F2Rqwn2oZQCeP2QF8yQaeWq5iDkgEVv3RTTg gMYAnigK4moNA6W7AUOJ7A/NDJnwR/sQ =4ytH -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Our beloved XML format doesn't parse; planet 11-11
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 15-11-10 23:38, Jon Burgess schreef: Have you changed anything recently? Like what, it was a fresh download, and I have never used PostGIS on my system. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkzhtyIACgkQYH1+F2Rqwn29nQCdE22v9PoeTYQFgrUW30BRG+vr lygAn2ppZP3lkDj3bCIBo1bAzTz2tvnm =NYm5 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Planet in PBF?
Is it possible to generate them and be placed on planet.openstreetmap.org ? Stefan Op 14 nov 2010 om 05:59 heeft Scott Crosby scro...@cs.rice.edu het volgende geschreven:\ On Thu, Nov 11, 2010 at 4:06 AM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, Did anyone already made an attempt to generate a (full) Planet PBF file? Yes. I have made several full planets when doing my size and speed benchmarks. Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Planet in PBF?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 14-11-10 23:13, Scott Crosby schreef: On Sun, Nov 14, 2010 at 4:54 AM, Stefan de Konink ste...@konink.de mailto:ste...@konink.de wrote: Is it possible to generate them and be placed on planet.openstreetmap.org http://planet.openstreetmap.org? I'm guessing that it could be done, but you need to ask whoever runs that server to do it. Hence my reply to this development mailinglist where everyone steps in when the topic is about development and licensing issues are strictly ignored. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkzgX34ACgkQYH1+F2Rqwn31qwCggTUOisI4Wt5yJSV7edfq38fL WnYAni8Iy66hq4y72+QhmMASLhvfUCI0 =EN96 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Planet in PBF?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, Did anyone already made an attempt to generate a (full) Planet PBF file? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkzbwBoACgkQYH1+F2Rqwn3e0wCfWuA2jubN3ksVkXVuLAIhgF7t SJwAoIk+J6J8dNi2QBsBnMlmN943IIuz =SOWt -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] PBF parsing frontend for osm2pgsql, tester wanted
On Mon, 1 Nov 2010, Dane Springmeyer wrote: +1 on pure C++. I assume we'd only need the protobuf C++ headers then, instead of both the c++ and c? Please team up with Mercator (Chris) because now we have C version, and C++ was already requested by many. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Mon, 25 Oct 2010, Chris Browet wrote: Sure. I don't think the C - C++ will be problematic. It might be, because a completely different library/code generation program is used. Anyway, I still need to look into that. Things that I must do for Merkaartor: - Take the logic out of main() - obviously. Maybe move the logic to a bool process_pbf(FILE* in) function. - Move the XML generation out, maybe replacing the #define's by some function pointers (possibly #ifdef'ed to keep the standalone pbf2osm as it is). I have been thinking about 'hooks' so the defines can become hooks you replace (So somesort of ifndef way.) Or just inline functions... If you'd accept some related patches, it would greatly improve re-usability and trackability of pbm2osm... Obviously I do ;) Just hadn't the time to integrate it yet. And busy day a head with flying ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
Thanks for contributing :) really gives stimulance to work more on it. Currently sitting on a couch at the Mentor Summit at Google. Some other patches came in as well,so i'll try to integrate everything after the show here :) Totally offtopic: everyone check out the MetaWritter GSoC project of Mapnik ;) STefan On Sun, 24 Oct 2010, Hartmut Holzgraefe wrote: On 09/24/2010 10:10 PM, Stefan de Konink wrote: As far as I can see with a quick look we are in a state of functional. [...] (you need to have protobuf-c installed, to compile it, see the Makefile for that) took a bit to figure out that things do not work the protobuf-c version that comes with Ubuntu Maverik. For this and other reasons i have taken the freedom to autotoolize things ... and while i was on it i also fixed most compiler warnings and tried to tweak performance a little bit, mostly by simplifying itoa() as it needs to work with base 10 only anyway, Perf. gain is about 5-10% ... My code is currently visible on http://github.com/hholzgra/hartmut-pbf2osm To build from a fresh checkout you now need to: ./autogen.sh ./configure make or just ./configure make when downloading the make dist tarball from http://php-baustelle.de/pbf2osm-0.11.tar.gz (the 0.11 as version number is rather arbitrary ...) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Mon, 25 Oct 2010, Chris Browet wrote: Would you mind if I re-use your code to implement a pbf reader in Merkaartor? Not at all, I just hope that everything that is coded now is correct, and you are able to track 'potential' further improvements. The current plan is also to equip mapnik with a pbf reader. (Mapnik devs here at the Mentor Summit...) but that does require a C++ approach, which might be beneficial for Merkaartor as well. So we could also team up rewriting the code using the C++ protobuf for integration in Merkaartor and Mapnik. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
Plan is first to release native support for osm2pgsql. Stefan Op 18 okt 2010 om 11:59 heeft Peter Körner osm-li...@mazdermind.de het volgende geschreven:\ Am 26.09.2010 17:14, schrieb Stefan de Konink: When this tool is polished enough (must include some bbox'ing then) we can think about osm2pbf. Are there any plans towards this? Will osm2pbf support the visible- flag? I tried to process the full-experimental dump into .osm.pbf using osmosis but the information about deleted elements gets lost through the osmosis pipeline. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Binary Format, Performance, pbf2osm
On Sun, 17 Oct 2010, Frederik Ramm wrote: The *** case couldn't be tested: Error unpacking PrimitiveBlock message. The offending file is at http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to check what's wrong with it. Gonna fix that :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Binary Format, Performance, pbf2osm
On Sun, 17 Oct 2010, Stefan de Konink wrote:
On Sun, 17 Oct 2010, Frederik Ramm wrote:
The *** case couldn't be tested: Error unpacking PrimitiveBlock message.
The offending file is at
http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to
check what's wrong with it.
Gonna fix that :)
message 'PrimitiveBlock': missing required field 'stringtable'
Error unpacking PrimitiveBlock message
(gdb) print *hmsg
$3 = {base = {descriptor = 0x406740, n_unknown_fields = 0,
unknown_fields = 0x0}, bbox = 0x0, n_required_features = 0,
required_features = 0x0, n_optional_features = 0, optional_features =
0x0,
writingprogram = 0x0, source = 0x0}
But lets look what is in the raw message:
(gdb) print bmsg.raw
$2 = {len = 56,
data = 0x60b550
\n\032\b\276\210\360\350B\020\340\220\361\227g\030\236\226\305\344\370\002
\340\347\351\223\340\002\\016OsmSchema-V0.6\\nDenseNodes\201\n\002}
Could it be an empty block?
Stefan
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Binary Format, Performance, pbf2osm
On Sun, 17 Oct 2010, Stefan de Konink wrote:
message 'PrimitiveBlock': missing required field 'stringtable'
Error unpacking PrimitiveBlock message
(gdb) print *hmsg
$3 = {base = {descriptor = 0x406740, n_unknown_fields = 0,
unknown_fields = 0x0}, bbox = 0x0, n_required_features = 0,
required_features = 0x0, n_optional_features = 0, optional_features = 0x0,
writingprogram = 0x0, source = 0x0}
But lets look what is in the raw message:
(gdb) print bmsg.raw
$2 = {len = 56,
data = 0x60b550
\n\032\b\276\210\360\350B\020\340\220\361\227g\030\236\226\305\344\370\002
\340\347\351\223\340\002\\016OsmSchema-V0.6\\nDenseNodes\201\n\002}
Could it be an empty block?
(It missed a required feature)
If we look at one block after it, which the error actually has:
(gdb) print *bmsg
$1 = {base = {descriptor = 0x4057e0, n_unknown_fields = 0,
unknown_fields = 0x0}, has_raw = 1, raw = {len = 141264,
data = 0x77eaf010 \n\261D\n}, has_raw_size = 0, raw_size = 0,
has_zlib_data = 0, zlib_data = {len = 0, data = 0x0}, has_lzma_data = 0,
lzma_data = {len = 0, data = 0x0}, has_bzip2_data = 0, bzip2_data = {
len = 0, data = 0x0}}
The block has raw_size = 0 (that will result in badness!)
Now the question is? I'm trusting raw_size (because it is equal for
uncompress and compressed data) and it means the output size. I see that
you have only set 'len' of the raw message. While it is set in the
compressed case. Why is it 0?
Stefan
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Binary Format, Performance, pbf2osm
On Sun, 17 Oct 2010, Frederik Ramm wrote: The *** case couldn't be tested: Error unpacking PrimitiveBlock message. The offending file is at http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to check what's wrong with it. I have now explicitly said raw_size in the uncompressed variant. That fixes this 'bug', so if you can give us a grain-of-salt timing that would be nice ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 17-10-10 19:42, Frederik Ramm schreef: (My dislike of git stems largely from ignorance and is likely to vanish over time. At the moment I have grown used to the one-stop-shop that is our SVN; with git I miss the option to say check out all OSM projects, as well as the option to commit to trunk for all OSM projects without first signing up to external services and/or find out who the maintainer is and ask them for permission.) Allow people to pull from you instead of push them your (unreviewed) code, as you do now with svn. Different mentality but generates a better quality project overall. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAky7N3sACgkQYH1+F2Rqwn1UgwCfReY56Z7I7rnxc/NXKxlditVI Rz4AmwVdg3G6H5UdoPI5Ln1UgRkSAd1O =XGsx -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Frederik, Op 17-10-10 20:11, Frederik Ramm schreef: (1) every project will have to have a maintainer who decideds what gets pulled into trunk - this introduces more formality than we have now; Maybe our data is informal. I rather not have code to be informal, directly the reason why I choose a GIT repository: people can branch my code create mess, fix stuff up and send it back. (2) as a committer, I want it to be *my* decision whether I commit something right away - in cases where I'm sure it is good - or whether I want to discuss with others first. That's in keeping with the spirit of OSM where you don't have to ask for permission to edit the map. If someone exposes peoples privacy, will his commits right be reverted? (3) as a user, I will have to hunt through mailing lists and whatnot and compile the cryptic URLs of private repositories to pull from, rather than having a one-stop shop. If it /is/ a one-stop shop, make svn accessible with the OSM credentials, just see what happens. ... git may offer more flexibility but compared with OSM where everyone gets SVN access...? So 'everyone' is not 'everyone' by default? But as I said - it doesn't have to be either-or, you can go ahead with the cutting edge development on git and we can every now and then pull something workable into SVN, with a big README that says please apply changes to git only. I thought by now there are svn2git bridges. So can we please not bikeshed about what kind of versioning system we use, instead ask ourselves why it took two weeks to get the serious bugs out of an app... because just nobody cared? And the point is, if I release the pbf version of your tool... when can we make sure that people start caring about it? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAky7QfIACgkQYH1+F2Rqwn3DnACgkpkKgBIEVJ3shVpayZWtx36t TSsAoJUaPIHWSSh7XRhE1YfAzjsX3MuF =Wbvl -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Binary Format, Performance, pbf2osm
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 17-10-10 22:13, Scott Crosby schreef: I left the raw_size unset, and you're getting the default value of 0. If this is by design, make it explicit on the wiki page (aka: this value is unset if no compression is used) I fixed it already... but I expected it would work different. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAky7ZesACgkQYH1+F2Rqwn2bPwCfbXIGT6awc8Y5qi93PDuPziXc eUEAn2WSm4XhTl3Xlf108yloXCVMkLHB =dqv5 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.
On Sat, 16 Oct 2010, Scott Crosby wrote: Also, can you give me the URL for the source code for pbf2osm to check that a few edge-cases are correctly handled? http://git.openstreetmap.nl/index.cgi/pbf2osm.git/ Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Editors: Actively removing created_by?
On Sun, 10 Oct 2010, Chris Browet wrote: I think it's desirable. They'll still be available in the object history, and the removal is So do I. I wonder why a bot hasn't been run to remove them all. I guess that would slim the planet measurably... And increase the database considerably. So thats why it won't hurt if editors remove it while something useful happens. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
Op 30-09-10 06:30, Scott Crosby schreef:
Also Osmosis currently implements deflate, not bzip2/lzma. (At the
time of writing I implemented bzip2 as well, but couldn't test it.)
If I can find a lzma/bzip2 java code, its ready to plug them in.
Although, I'm thinking that with the cost of bzip2 decompression, the
only interesting case is lzma.
http://www.7-zip.org/sdk.html
I would be happy to help in the design and algorithms of a geometric
index, but I don't have the time to program it. Are you interested in
coding this? If done properly, geometric sorting code can do much much
more than make a planet.osm.pbf indexable.
Always interesting in coding stuff ;)
They can make very cheap tileservers. PrimitiveBlocks are independently
decodable blobs; they can be stored in a database. Given a bbox query,
select out all of the blobs that intersect it, concatenate them
together, add a header, and thats a conforming *.osm.pbf file. Depending
on the precision used and whether metadata is kept, that entire database
can be cached into 6-10GB of RAM.
I would like to see this happen.
We should pull in the guy that did this for the BZ2 planet.
This may be a new way to distribute the planet: A set of PrimitiveBlock
tiles sitting in an Sqlite database. Actually, now that I think of
it, this may be the superior approach compared to using my hooks to hack
an index into the *.osm.pbf format.
- allow the use of the bbox
What exactly do you mean by this: ?
Filter output by applying a bbox (the expensive way). Extract nodes
within a bbox, sort them on ID, extract all ways, validate per item if
any of the nd's are selected. Append remaining nodes to a miss list.
Same operation for relations. Run lookup for missing ways, fetch missing
node list.
+ based on the index
The 'open index' that is not implemented should be implemented.
And this: ?
+ based on geos like solution
Allow the user to make non-rectangle selections.
My personal interest is going for output that support output that
can be used to 'copy into' data into a database. And extend the
current protocol for diff support. (Hence: replacing osmsucker-ykw)
What kinds of extensions are you thinking of?
What we discussed {create,modify,delete}.
Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEAREKAAYFAkykW/EACgkQYH1+F2Rqwn2wxwCghZVt891iuKM2n+O/oJjdBhbX
jUMAoIJhCbBsQNr/NiVAoDU+ywnDeI3n
=EaGx
-END PGP SIGNATURE-
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Thu, 30 Sep 2010, Scott Crosby wrote: I proposed exactly this for the mkgmap splitter. If you're going to do this, can I propose a tweak where you can output thousands of files simultaneously? The differences are minor: Instead of tracking if a node/way/relation was output or was missed with two bitsets, track which areas it has been dumped to with two multimaps from the ID to the list of areas it was output to or the list of areas in which it was missed. How would you a priori know if a node is in a certain area if you haven't observed its locatio or trace it? As the typical keycount in the multimap is low, you can use the trick I used a few weeks ago in the splitter: build a multimap by layering a set of individual hash tables, one for the first value of each key, a second for the second value of each key (if any), etc. Use sparse hash tables ( http://code.google.com/p/google-sparsehash/) for the first few layers, and std::map for the remaining layers. Bleggg... C++ :r + based on the index The 'open index' that is not implemented should be implemented. Ok. What kinds of things might we want to index? BBOX? Count of nodes/ways/relations in that block? What else? I would find it very interesting if different types of output could be exported individually. For example being context aware. Some data is landuse, I don't need landuse for routing, so it might be exported in a completely different part of the pbf. So if the format would be descriptive about 'exclusive roads' that might also help the application that uses the data to extract or leave the set. I don't think that counts are useful. The mbr is. Ok, Is XML's gzipped size or parsing speed a bottleneck for storing or processing changes? I'd be happy to offer suggestions on the protocol buffer architecture. From what I observe now the bottleneck seems to be actually protocol buffers, while my output code can become slightly faster. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Thu, 30 Sep 2010, Scott Crosby wrote: From what I observe now the bottleneck seems to be actually protocol buffers, while my output code can become slightly faster. This would imply that doing binary changesets isn't a critical necessity. You asked where the bottleneck is, for XML generation. Not for parsing XML to something useful. PBF is very nicely structured... but first things first. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Thu, 30 Sep 2010, Frederik Ramm wrote: Stefan de Konink wrote: When this tool is polished enough (must include some bbox'ing then) we can think about osm2pbf. Speaking of polished: The program currently produces invalid XML because and are not escaped, leading to lines like Yes, Roeland pointed that out as well yesterday. We have discussed an escape table. Maybe first parsing the entire string table, alternatively doing it for each instance. Other than that, it runs about twice as fast as Osmosis so that's a good sign. What does the README mean when it says: At the time of writing it can only handle dense nodes? ...that the README is already outdated ;) But what I already discussed with Scott, we *need* a good 'has everything' PBF file. Something that can test a parser and has expected output. For example Osmosis only generates dense nodes, not the 'other notation'. Also Osmosis currently implements deflate, not bzip2/lzma. (At the time of writing I implemented bzip2 as well, but couldn't test it.) There are a few things on the todo before going for the osm2pbf myself: - xml escaping - mmap input - lzma - allow the use of the bbox + based on the index + based on geos like solution Roeland wanted to go for a library so other tools could call getNextNode() etc. without fiddling with the internal structures. My personal interest is going for output that support output that can be used to 'copy into' data into a database. And extend the current protocol for diff support. (Hence: replacing osmsucker-ykw) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Wed, 29 Sep 2010, Anthony wrote: In addition to and , you need to escape . planet.c also escapes . It uses character references for each (quot;, amp;, lt;, and gt;). planet.c also escapes carriage return, line feed, and tab, as #13;, #10;, and #9;. AFAICT it is legal to include these unescaped (though it would be nice to escape at least line feeds to make it easier on fast, non-XML-compliant parsers). What is currently in git escapes the first 5 cases. But doing the other few shouldn't be a big issue. I implemented an almost printf free version, but I have some problems getting good benchmarks between versions, need to find out where some massive overhead comes from. (Some person schedules bulk processing every night... while I'm coding... no clue why he can't schedule them when people are not coding... *g*) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Peter, Op 26-09-10 17:00, Peter Körner schreef: Am 24.09.2010 22:10, schrieb Stefan de Konink: Or the (static) 64bit version: http://mirror.openstreetmap.nl/pbf2osm64 p...@maps:~$ pv -cN 'pbf-input' /osm/data/planet-extract/hessen.osm.pbf | ./pbf2osm64 | pv -cN 'xml-output' /osm/data/planet-extract/hessen.osm.xml pbf-input: 43.1MB 0:00:13 [3.14MB/s] [==] 100% xml-output: 751MB 0:00:13 [54.7MB/s] [ = ] Thanks for that benchmark. Roeland and I are probably going to make the output as 'unformatted' where possible. And the file reader (currently you see a stream reader) will be based on memory mapping, thus less data to allocate, more fun on the OS side. When this tool is polished enough (must include some bbox'ing then) we can think about osm2pbf. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkyfY1UACgkQYH1+F2Rqwn2dkQCcC63xWdqltX/SQSNF684nOhrk VBEAninN7YEJ+Qgz/5qhO6O64orRwHbF =ViKm -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, As far as I can see with a quick look we are in a state of functional. Tweaking will be done of course, especially because this version is only a pipe. cat test.pbf | pbf2osm output.osm You can check it out at: http://git.openstreet.nl/pbf2osm.git (you need to have protobuf-c installed, to compile it, see the Makefile for that) Or the (static) 64bit version: http://mirror.openstreetmap.nl/pbf2osm64 Feedback, comments, segmentation faults, can be e-mailed directly. This is not the final version, but it is something that should give correct output. File loading (and mmaping) will be done in the next release. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkydBbwACgkQYH1+F2Rqwn2HXQCeLWw4kJsWg/RMHsgAnYkWyvAX q5MAoJOzvnrw9onJOTRaIfDdR9f/H3Uj =v/zp -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 23:20, Anthony schreef: Some quick notes... Had to create ../OSM-binary/src Had to download the .proto files and move them into ../OSM-binary/src Had to make ../external and ../external/lib Make sure you have the newest version of protobuf-c. I guess git submodule update should actually do the trick for you. After that, I was able to compile everything. The 64bit compiled version worked out of the gate. Still haven't gone about comparing the output. Ok, if we need to move some tags around to make it easier to diff, that is obviously possible. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkydGCkACgkQYH1+F2Rqwn2kdQCePoAjYejSEgHjRWEXAqYz1EL6 vaUAnj8CqYKB1BGdzkqpdf3JkYZ2CpzR =VfNN -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 As annouced some days ago I would start with the development if nobody stepped up before Friday. Nobody did, so I will do so. Language of constraint is C, we will most likely not use any external dependancies. So porting to different platforms including Windows should be trivial. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkybzzcACgkQYH1+F2Rqwn2a7ACeJsF2W+acO267enaHSTqkZMGD qC8AnAtLNbcwujGnt78CefxZH2nrUSiI =lmqw -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 02:03, Anthony schreef: On Thu, Sep 23, 2010 at 6:18 PM, Frederik Ramm frede...@remote.org wrote: Language of constraint is C, we will most likely not use any external dependancies. Well you won't get around the Google Protobuf stuff of course! The Google Protobuf stuff is in C++. There is a nice C variant of it too. http://code.google.com/p/protobuf-c/ But I find things like the following more interesting: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Since I don't get a decompression error, I wonder if osmosis is using a different proto than I am doing ;) Questions ;) Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycAooACgkQYH1+F2Rqwn2I7gCfd2WmfCRjKApOvTX3SXOOoYr+ aKIAmwZNyr2a0nWy32F5j+O6kz5M3e1t =zs57 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 03:44, Stefan de Konink schreef: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Never mind, I expected protobufs to be more robust ;) It isn't. Fixed. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycBoIACgkQYH1+F2Rqwn08HQCggUlcXRfJxCL4ayfRrie+Bepp HWgAni/WQrEoBngiE0TcPWqEp1WCmd9e =Ce/d -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 04:26, Scott Crosby schreef: What was the problem and the fix? The problem was that I assumed that after a succesful inflate nothing in my inflate buffer could have gone wrong. But I was actually placed on the end of that buffer. So basically protobuf pretends everything is 'ok' and just starts parsing. So the fix was just nicely to pass the start of the inflated data. I was coding quite defensive, but it seems protobuf itself still 'eats' memory. Given the presented 'max 64MB limits', not really an issue, but it is just duplicating a lot of memory, instead of using pointers to the input arrays. For whats worth, currently I'm able to decode the 'fileformat', doing deflate decompression. So still some stuff to go, but I understand the basics now... so I'm a happy farmer. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycE98ACgkQYH1+F2Rqwn2svwCghj5x+u77Dg6ZIO1Fb9IdP/4l jMUAn1Irj/uLFzdESAZMQ/MnJ8X/oQHO =9QVf -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 04:58, Stefan de Konink schreef: For whats worth, currently I'm able to decode the 'fileformat', doing deflate decompression. So still some stuff to go, but I understand the basics now... so I'm a happy farmer. Small status update, Dense Nodes implemented. Including XML rendering. Normal nodes, ways and relations remaining. Thingie will be finished later today. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycMT0ACgkQYH1+F2Rqwn1krQCfVRAmO52YMg+RWvKZ1iuRldQB wPoAoI3t1O7h2LjdwpHj+C7DgCTwJGhD =38Kl -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 22-09-10 14:34, Frederik Ramm schreef: I'm sure we'll get a pure C/C++ implementation sooner or later; even so, there certainly will be people who don't feel like installing it. If noone steps up for the reference implementation before Friday I'll write one. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkyZ+skACgkQYH1+F2Rqwn0C5gCeJct2LDyH9JgAj/p3zwyBgaQS +ncAoIM44Bh/rFKFGfWwGWihp7KUZPIG =vN++ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
On Wed, 22 Sep 2010, Frederik Ramm wrote: My guess is that 95% of people who use these files process them either with osmosis, or mkgmap, or osm2pgsql. Don't make the same mistake as with disabling the gazetteer. Legacy can be useful, is there currently a plain C implementation with a from/to xml? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] PostgreSQL 9.0?
On Sun, 19 Sep 2010, Ian Dees wrote: Has anyone started playing with Postgres 9.0 and OSM data? I'm particularly interested in the streaming replication system they've introduced. Only tested how PostGIS was working, other than 'it works' with some patches, not really having input. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] size of apidb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 27-08-10 20:50, Marco Lechner - FOSSGIS e.V. schreef: I'm confused and asking for enlightenment. Not all data might as well aligned as you might think ;) A lot of deletes and updates in a table can make it pretty sparse. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkx4HYYACgkQYH1+F2Rqwn0UOgCeJddCAKIc72Ii2zMg4ac4T6uE C9wAoIr5uDZFkc1gkN1Tc6RoxXYWmA5k =B0jC -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to use XAPI
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 15-08-10 16:31, bernhard zwischenbrugger schreef: Hi all My next project is an XAPI map. You can see a prototype at: http://www.khtml.org/osm/v0.76/examples/xapi.html It is possible to make a global search for power_source=nuclear But a global search for amenity=restaurant would bring too much results and it would produce too much load for the db. My question: Is there a possiblity to bring this feature to the normal users? Is there an undocumented limit parameter to reduce the output to a maximum number of results? Any other ideas how to enable this feature? Using the default XAPI you can also bbox the request :) bbox it to 110% of the viewport at a certain zoomlevel and you probably have a winner. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkxn/tQACgkQYH1+F2Rqwn0WwACdGZpD5XCMreQcCpr4GD4Tqdwl W0YAnRQDyrXMjbAcr2JeJqzh7yXNemrY =GJoI -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] CC-BY-SA datasources, display warning in editor?
On Wed, 4 Aug 2010, Frederik Ramm wrote: Otherwise people might just put in a lot of work for nothing. Obviously not true, since the data will be available in the CC-BY-SA output. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New full history dump available
On Mon, 2 Aug 2010, Lars Francke wrote: Matt Amos was so nice to run the history export again. The result is available here: http://planet.openstreetmap.org/full-experimental/full-planet-100801.osm.bz2 and it's grown from 13 GB in February to 17 GB. The regular planet has grown from 8 to 10 GB in the same time. Have fun and it would always be interesting to know if you're using it, having problems with it and what you're using it for. Could someone also put this out as a torrent? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Sun, 1 Aug 2010, Frederik Ramm wrote: A lot of time is spent just reading from, and writing to, disk and parsing XML. Running the whole thing with .gz files doesn't make a big difference - saves some disk i/o, adds some CPU time, doesn't change XML parsing overhead. I'm sorry but the parsing overhead from Java or libXML basically a known slowless factor. MSXML, pre/post plane parsing or even custom readers are not slow, and only limited to the disk. So the binary format, per se, is only faster because: - smaller filesize = less io - encoding: no xml rewriting Anything else is currently available using for example osmsucker.c, obviously not using an XML parser because all input is structured. If the binary format can pack our doubles (lat/lon), integers (version/ids) and makes strings available in UTF-8, that skips CPU and IO overhead. But makes the data not human readable. I can totally live with that, and I hope the API protocol also gets protocol buffers. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Sun, 1 Aug 2010, Anthony wrote: On Sun, Aug 1, 2010 at 6:00 PM, Stefan de Konink ste...@konink.de wrote: If the binary format can pack our doubles (lat/lon) lat/lon is stored as a double? I always use an int (and divide/multiply by 1000). http://wiki.openstreetmap.org/wiki/Database_schema Yeah, OSM seems to be doing the same thing. OSM uses too many digits for 16bit numerics anyway ;) But I do hope that you don't expect your geometry engine to store this in an int ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Creation of dataset for import
On Wed, 21 Jul 2010, 80n wrote: Right now, it seems to me that tracing twice might be the most effective solution. The simple solution obviously would generate SQL from a diff file, that can by pass the api. This could be relatively fast. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Creation of dataset for import
On Wed, 21 Jul 2010, 80n wrote: High performance is not a requirement. Fidelity of merging edits that are potentially conflicting is important. There will be some divergence between the OSM dataset and the once that is being created. Actually it is, the faster the transactions can be pushed through, the lower the chance of collissions will be. If this file takes you a week to upload by the api, the consequences are easy to see. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Creation of dataset for import
On Wed, 21 Jul 2010, 80n wrote: It's going to take several months to prepare *before* it even meets OSM, so speed really isn't a requirement. So what you actually are looking for is a tool that can interactively fastforward changes? As you would have with a source code versioning system? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Has anyone got a cheat sheet for forking OSM at all?
On Tue, 20 Jul 2010, John Smith wrote: I've seen schema and such on the wiki, but nothing putting it all together like some of the mapnik/mod_tile/etc tutorials. If nothing exists I'll start documenting it as I go... I wrote a complete read/write API in C. http://wiki.openstreetmap.org/wiki/Cherokee/MonetDB_Handler_OSM Currently working on getting it 'more' compatible with 0.6, but the implementation is fully 0.5 compatible and includes XAPI. Because I'm mentoring Mitja that is trying to create a 'better' XAPI I'll do some efforts to get this 'old code' to work with the newer editors. The code what you see is basically a drop in placement for the Cherokee Webserver. Such as mod_tile is for apache. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
