Re: [OSM-dev] Binary Format, Performance, osm2pbf

2010-10-17 Thread Scott Crosby
On Fri, Oct 15, 2010 at 8:56 AM, Frederik Ramm frede...@remote.org wrote: Hi, a few weeks ago when I announced that I am planning to rely on the new binary format in the future, I caught some flak for claiming that an .osm.pbf was not only faster to produce, parse, and transmit than a .bz2

Re: [OSM-dev] Disallowing certain characters in tag keys

2010-10-17 Thread Ulf Lamping
Am 16.10.2010 20:44, schrieb Jochen Topf: Hi! I am currently fighting some issues where tags with strange characters in them need to be represented in a URL for Taginfo. Lots of other websites probably will have similar issues. Characters like /, ?,, etc. have special meaning in URLs so if they

Re: [OSM-dev] Disallowing certain characters in tag keys

2010-10-17 Thread Tony Morris
The problem with disallowing special characters is that you close the door. Software writers will then write software that depends on them not being there (or not caring which is probably the common case today). If we later find out that - for whatever reasons - we want to use one of those

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Frederik Ramm
Hi, updated according to Scott's request (and subject fixed), comparison using the OSM Bavaria extract: File sizes: .osm 2462778377 .osm.pbf 134579577 (with compression=deflate and lossless) .osm.pbf 255103238 (uncompressed) .osm.bz2 223006298 (-2) .osm.gz 280224099

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Stefan de Konink
On Sun, 17 Oct 2010, Frederik Ramm wrote: The *** case couldn't be tested: Error unpacking PrimitiveBlock message. The offending file is at http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to check what's wrong with it. Gonna fix that :) Stefan

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Stefan de Konink
On Sun, 17 Oct 2010, Stefan de Konink wrote: On Sun, 17 Oct 2010, Frederik Ramm wrote: The *** case couldn't be tested: Error unpacking PrimitiveBlock message. The offending file is at http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to check what's wrong with it.

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Stefan de Konink
On Sun, 17 Oct 2010, Stefan de Konink wrote: message 'PrimitiveBlock': missing required field 'stringtable' Error unpacking PrimitiveBlock message (gdb) print *hmsg $3 = {base = {descriptor = 0x406740, n_unknown_fields = 0, unknown_fields = 0x0}, bbox = 0x0, n_required_features = 0,

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Frederik Ramm
Scott, Thanks, I enjoyed looking at your code and style of coding. The code is Stefan's. I only wrote about an idea I had to wrap it up into something more convenient - but I wasn't involved until now. Bye Frederik ___ dev mailing list

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Stefan de Konink
On Sun, 17 Oct 2010, Frederik Ramm wrote: The *** case couldn't be tested: Error unpacking PrimitiveBlock message. The offending file is at http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to check what's wrong with it. I have now explicitly said raw_size in the

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Frederik Ramm
Hi, updated after Stefan's bugfix. Only the line marked *** is new. File sizes: .osm 2462778377 .osm.pbf 134579577 (with compression=deflate and lossless) .osm.pbf 255103238 (uncompressed) .osm.bz2 223006298 (-2) .osm.gz 280224099 Decompression (saving to .osm.xml): from .osm.bz2

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 6:36 AM, Frederik Ramm frede...@remote.org wrote: Scott, Thanks, I enjoyed looking at your code and style of coding. The code is Stefan's. I only wrote about an idea I had to wrap it up into something more convenient - but I wasn't involved until now. I think

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Frederik Ramm
Hi, Scott Crosby wrote: I think the perfect permanent place for Stefan's implementation, and your idea of wrapping it up as a debian package, is with the http://github.com/scrosby/OSM-binary main repository. Is there an OSM GIT I can clone it to, or should it really be put into OSM SVN? I

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Stefan de Konink
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 17-10-10 19:42, Frederik Ramm schreef: (My dislike of git stems largely from ignorance and is likely to vanish over time. At the moment I have grown used to the one-stop-shop that is our SVN; with git I miss the option to say check out all OSM

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Frederik Ramm
Hi, Stefan de Konink wrote: Allow people to pull from you instead of push them your (unreviewed) code, as you do now with svn. Different mentality but generates a better quality project overall. Not what I want, because (1) every project will have to have a maintainer who decideds what gets

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Stefan de Konink
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Frederik, Op 17-10-10 20:11, Frederik Ramm schreef: (1) every project will have to have a maintainer who decideds what gets pulled into trunk - this introduces more formality than we have now; Maybe our data is informal. I rather not have

Re: [OSM-dev] Disallowing certain characters in tag keys

2010-10-17 Thread Ulf Lamping
Am 16.10.2010 22:31, schrieb Andreas Kalsch: I agree with whitespace - this can be very confusing = To add: Make keys lowercase (or even remove diacritics), because keys are always simple names. I've added a Character section to the page:

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Frederik Ramm
Hi, Stefan de Konink wrote: (2) as a committer, I want it to be *my* decision whether I commit something right away - in cases where I'm sure it is good - or whether I want to discuss with others first. That's in keeping with the spirit of OSM where you don't have to ask for permission to edit

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 12:42 PM, Frederik Ramm frede...@remote.org wrote: Hi, Scott Crosby wrote: I think the perfect permanent place for Stefan's implementation, and your idea of wrapping it up as a debian package, is with the http://github.com/scrosby/OSM-binary main repository. Is

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
Not what I want, because (1) every project will have to have a maintainer who decideds what gets pulled into trunk - this introduces more formality than we have now; (2) as a committer, I want it to be *my* decision whether I commit something right away - in cases where I'm sure it is good

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Greg Troxel
Frederik Ramm frede...@remote.org writes: I've been the chief SCM crank at work for close to 15 years. We used CVS exclusively for a long time, and I didn't allow people who said foo is shiny this week; we should switch to change our world because none of them could answer the challenge of

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 2:40 PM, Greg Troxel g...@ir.bbn.com wrote: I think what's going on now is that we have a svn repo that git weenies don't particularly care for, and no git repo support on osm servers, and that people are therefore basically setting up git repos elsewhere for various

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Frederik Ramm
Greg, nice explanation; you provided a background to some things that until now were only a hunch for me. Greg Troxel wrote: I can certainly see the point of avoiding free-as-in-beer services with non-free-as-in-speech toolsets and changeable procedures. But I think that's orthogonal to

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 6:34 AM, Stefan de Konink ste...@konink.de wrote: On Sun, 17 Oct 2010, Stefan de Konink wrote: message 'PrimitiveBlock': missing required field 'stringtable' Error unpacking PrimitiveBlock message (gdb) print *hmsg $3 = {base = {descriptor = 0x406740,

Re: [OSM-dev] Disallowing certain characters in tag keys

2010-10-17 Thread Anthony
On Sat, Oct 16, 2010 at 2:44 PM, Jochen Topf joc...@remote.org wrote: Technically this would mean changing the API to check for those characters, removing any that are already in the database (can be done with normal manual edits because there are so few cases) and adding checks to the

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Stefan de Konink
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 17-10-10 22:13, Scott Crosby schreef: I left the raw_size unset, and you're getting the default value of 0. If this is by design, make it explicit on the wiki page (aka: this value is unset if no compression is used) I fixed it already... but

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Greg Troxel
Scott Crosby scro...@cs.rice.edu writes: On Sun, Oct 17, 2010 at 2:40 PM, Greg Troxel g...@ir.bbn.com wrote: I think what's going on now is that we have a svn repo that git weenies don't particularly care for, and no git repo support on osm servers, and that people are therefore basically

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Greg Troxel
Frederik Ramm frede...@remote.org writes: OSM already has a git repository but this is currently only used for the rails port - where I have to admit it makes some sense, as the rails port software happens to be more controlled than other stuff in that the admins must closely watch what gets