Re: [OSM-dev] Timestamp in PBF files

2012-12-04 Thread Scott Crosby
On Tue, Dec 4, 2012 at 2:12 AM, Frederik Ramm frede...@remote.org wrote: Hi, On 12/04/12 07:50, Jochen Topf wrote: That still isn't specific to Osmosis. Somebody else could implement this algorithm. Markus seems to have done so, albeit a bit differently. The algorithm should be documented

Re: [OSM-dev] Timestamp in PBF files

2012-11-24 Thread Scott Crosby
On Fri, Nov 23, 2012 at 5:03 AM, mar...@gmx.eu wrote: Hi Scott, in brief to the 1-degrees granularity: 1. Do whole processing in 64 bit: This would mean to need much more RAM space when processing ways' coordinates. We should not do this unless this granularity is really required. If

Re: [OSM-dev] Timestamp in PBF files

2012-11-22 Thread Scott Crosby
And we have changed the PBF format before and are in the process of changing it again, so it is not such a big deal to add support for these things later if they are actually needed. One of my goals was to reduce breaking changes, or making files that a program thinks it can read, but can't

Re: [OSM-dev] Timestamp in PBF files

2012-11-21 Thread Scott Crosby
On Wed, Nov 21, 2012 at 3:46 AM, Jochen Topf joc...@remote.org wrote: On Tue, Nov 20, 2012 at 09:17:59PM -0600, Scott Crosby wrote: Not quite. The granularity of timestamps can go down to the milliseconds. https://github.com/DennisOSRM/OSM-binary/blob/master/src/osmformat.proto#L96 Ugh

Re: [OSM-dev] Timestamp in PBF files

2012-11-21 Thread Scott Crosby
On Wed, Nov 21, 2012 at 5:26 AM, Frederik Ramm frede...@remote.org wrote: Hi, On 11/21/2012 11:50 AM, mar...@gmx.eu wrote: OK, this seems to be consensual: PBF id 18 in the header block for a signed int UNIX timestamp value. In both his messages, Scott had suggested PBF id 18 for a

Re: [OSM-dev] Timestamp in PBF files

2012-11-20 Thread Scott Crosby
On Tue, Nov 20, 2012 at 1:09 PM, Jochen Topf joc...@remote.org wrote: On Tue, Nov 20, 2012 at 06:57:50PM +0100, Stephan Knauss wrote: Frederik Ramm writes: I really don't mind *how* it's done but I would really love to have one agreed way to place a timestamp in a PBF instead of

Re: [OSM-dev] Timestamp in PBF files

2012-11-20 Thread Scott Crosby
Idea, why not put the entire state.txt file into the OSMHeader block? /* Contains the file header. */ message HeaderBlock { optional HeaderBBox bbox = 1; /* Additional tags to aid in parsing this dataset */ repeated string required_features = 4; repeated string optional_features = 5;

Re: [OSM-dev] Timestamp in PBF files

2012-11-20 Thread Scott Crosby
On Tue, Nov 20, 2012 at 6:51 AM, Frederik Ramm frede...@remote.org wrote: Hi, (message to dev list but explicitly Cc'ing Brett and Scott because I don't know if they follow dev) about a year ago, Marqqs tried to have a discussion on how to add timestamps to PBF files and hardly anyone

[osmosis-dev] Support for reading and writing PBF files to/from STDIN, STDOUT, or named pipes.

2011-11-08 Thread Scott Crosby
My fork of osmosis at https://github.com/scrosby/osmosis supports reading and writing PBF files from pipes. You can access this by either using named pipes or by using the magic filename '-' to denote STDIN/STDOUT. Please test this functionality and report any problems. Scott

Re: [OSM-dev] Release of osmpbf version 1.2.1

2011-10-31 Thread Scott Crosby
On Mon, Oct 31, 2011 at 9:31 AM, Jochen Topf joc...@remote.org wrote: On Mon, Oct 31, 2011 at 10:31:11AM +0100, mar...@gmx.eu wrote: * Includes support for visible flag on OSM objects. This allows PBF to   handle OSM history files. Great! Thank you - and Jochen. Could you please, if

[OSM-dev] Release of osmpbf version 1.2.1

2011-10-30 Thread Scott Crosby
The changes to this version are pulling Jochen Topf's branch from github (vesion 1.1.1j2). * Debian packaging * Switched to the lite (non-introspective) version of protocol buffers for a smaller library. * Includes support for visible flag on OSM objects. This allows PBF to handle OSM history

Re: [OSM-dev] osm2pgsql for 64-bit IDs

2011-06-05 Thread Scott Crosby
On Sun, Jun 5, 2011 at 8:41 AM, Anthony o...@inbox.org wrote: On Sun, Jun 5, 2011 at 8:39 AM, Frederik Ramm frede...@remote.org wrote: Linestrings with internal geometries are very unlikely to happen, for a number of reasons, one of them being topology. That's too bad.  More than 90% of node

Re: [OSM-dev] osm2pgsql for 64-bit IDs

2011-05-23 Thread Scott Crosby
On Sat, May 21, 2011 at 9:52 AM, Jochen Topf joc...@remote.org wrote: If we use unsigned ints we have some more time. Problematic would only be a few cases where negative IDs are currently used (like in JOSM for data thats not yet uploaded to the server). But it seems wasteful to me, to go to

Re: [OSM-dev] osm2pgsql for 64-bit IDs

2011-05-23 Thread Scott Crosby
On Sat, May 21, 2011 at 10:18 AM, Stefan de Konink ste...@konink.de wrote: Don't forget the binary format (PBF) also was basically twisted around to accommodate for negative numbers. Your suggestion makes sense, there shouldn't be negative IDs. And additional making the value NOT NULL in the

Re: [OSM-dev] Filename extensions for OSM files

2011-05-11 Thread Scott Crosby
On Wed, May 11, 2011 at 6:48 AM, Peter Körner osm-li...@mazdermind.de wrote: Am 11.05.2011 13:42, schrieb Frederik Ramm: Hi, On 05/11/11 13:23, Peter Körner wrote: that way it is useful for a program to know, if a file fed into it via stdin will be processed fine right from the start.

Re: [OSM-dev] PBF for History-Files (was: visible-Flag in PBF)

2011-05-11 Thread Scott Crosby
On Tue, May 10, 2011 at 11:55 AM, Jochen Topf joc...@remote.org wrote: On Mon, May 09, 2011 at 05:21:40PM -0500, Scott Crosby wrote: On May 8, 2011 5:21 AM, Jochen Topf joc...@remote.org wrote: So I propose the following: 1. Add the visible flag as I have already proposed

Re: [OSM-dev] Ordering OSM-Files (was: visible-Flag in PBF)

2011-05-11 Thread Scott Crosby
On Tue, May 10, 2011 at 11:12 AM, Jochen Topf joc...@remote.org wrote: On Mon, May 09, 2011 at 05:41:19PM -0500, Scott Crosby wrote: Yes. How about the following optional feature flags to indicate sort order:  Sort.NodeWayRelationThenID      --- Because this is the default order of the planet

Re: [OSM-dev] Ordering OSM-Files (was: visible-Flag in PBF)

2011-05-09 Thread Scott Crosby
On Sun, May 8, 2011 at 5:19 AM, Jochen Topf joc...@remote.org wrote: One point that came up in the PBF discussion was the questions of ordering OSM files. Lets try to analyze that and try to separate the different issues: 1. Order of object types OSM files contain three different kinds of

Re: [OSM-dev] visible-Flag in PBF

2011-05-07 Thread Scott Crosby
On Sat, May 7, 2011 at 1:25 PM, Christian Vetter veaac.fdi...@gmail.com wrote: Hi, On Sat, May 7, 2011 at 5:46 AM, Scott Crosby sc...@sacrosby.com wrote: On Fri, May 6, 2011 at 9:24 PM, Christian Vetter veaac.fdi...@gmail.com wrote: With regard to LZMA: I have some C++ code lying around

Re: [OSM-dev] visible-Flag in PBF

2011-05-07 Thread Scott Crosby
On Sat, May 7, 2011 at 2:06 PM, Christian Vetter veaac.fdi...@gmail.com wrote: Hi, On Sat, May 7, 2011 at 5:46 AM, Scott Crosby sc...@sacrosby.com wrote: One idea: Each BlobHeader includes an 'indexdata' field. We could store a protocol buffer message there. It is available without parsing

Re: [OSM-dev] visible-Flag in PBF

2011-05-07 Thread Scott Crosby
On Sat, May 7, 2011 at 3:01 PM, Christian Vetter veaac.fdi...@gmail.com wrote: On Sat, May 7, 2011 at 9:23 PM, Scott Crosby sc...@sacrosby.com wrote: On Sat, May 7, 2011 at 1:25 PM, Christian Vetter veaac.fdi...@gmail.com wrote: Hi, On Sat, May 7, 2011 at 5:46 AM, Scott Crosby sc

Re: [OSM-dev] visible-Flag in PBF

2011-05-07 Thread Scott Crosby
On Sat, May 7, 2011 at 4:04 PM, Christian Vetter veaac.fdi...@gmail.com wrote: There are about 80k blobs. If 1-byte tags are used for the counts, overhead is:9 bytes each:   2 bytes indexdata taglength in the BlobHeader   3*1 bytes (tags for 3 fields)   2*1 bytes (varint count for N==0)  

Re: [OSM-dev] visible-Flag in PBF

2011-05-06 Thread Scott Crosby
On Fri, May 6, 2011 at 8:14 AM, Christian Vetter veaac.fdi...@gmail.com wrote: Hi, When you assign ids to the new fields do take care to assign small ids. Every id 32 can be encoded in one byte by protobuf. Every tag 16 can be encoded in one byte, but that doesn't matter in this case. An

Re: [OSM-dev] visible-Flag in PBF

2011-05-06 Thread Scott Crosby
On Thu, May 5, 2011 at 9:42 AM, Peter Körner osm-li...@mazdermind.de wrote: Hi I'm working heavily with the full-history-dumps. Lately I wrote a splitter [1] that allows splitting the full-history-dump into smaller extracts and I plan to publish some of the extracts on a GWDG mirror. Right

Re: [OSM-dev] visible-Flag in PBF

2011-05-06 Thread Scott Crosby
On Fri, May 6, 2011 at 9:24 PM, Christian Vetter veaac.fdi...@gmail.com wrote: On Sat, May 7, 2011 at 3:58 AM, Scott Crosby sc...@sacrosby.com wrote: Yes, but via a different mechanism. Create a new blob type: 'OSMInvisibleHistoryData', it will contain a serialized ordinary PrimitiveBlock

Re: [OSM-dev] visible-Flag in PBF

2011-05-06 Thread Scott Crosby
On Fri, May 6, 2011 at 8:58 PM, Scott Crosby sc...@sacrosby.com wrote: Yes. If you want to create an incompatable hpbf (History PBF) format that is very similar to the existing PBF format. Encode the visible flag into the protocol buffers along the lines of Jochen's suggested patch, with a few

Re: [OSM-dev] Using osmosis to generate a full-planet.osm file

2011-04-06 Thread Scott Crosby
On Mon, Apr 4, 2011 at 12:14 PM, Eric Wolf ebw...@gmail.com wrote: I assume osmosis is used to generate the full-planet.osm file. I want to generate a similar file from my OSM server. What is the proper set of parameters to do this? I've tried this with a failure: osmosis --read-apidb-change

Re: [osmosis-dev] Cannot represent -1 as a char.

2011-03-08 Thread Scott Crosby
Brett, do you have any advice? In OsmosisBinaryParser, I'm using NOCHANGESET and NOVERSION when there's omitted metadata. Currently both are -1. The bug is because another serializer in osmosis is complaining that the version number is negative. We can set NOVERSION to 0, make that code more

Re: [osmosis-dev] Cannot represent -1 as a char.

2011-03-06 Thread Scott Crosby
then I get the exception: SEVERE: Thread for task 1-read-pbf failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent -1 as a char.    at org.openstreetmap.osmosis.core.util.IntAsChar.intToChar(IntAsChar.java:35)    at

Re: [OSM-dev] .pbf files

2011-01-31 Thread Scott Crosby
Osmosis uses densenode by default, but can optionally use node messages. I know of no other implementations of a pbf file writer. On Jan 31, 2011 8:49 AM, Chris Hill o...@raggedred.net wrote: I have been experimenting with the binary format .pbf files. Does anyone who creates .pbf files use the

Re: [OSM-dev] osmpbf v1.1.1 jar released. It should fix the PBF-on-windows issues.

2011-01-26 Thread Scott Crosby
On Tue, Jan 25, 2011 at 11:37 PM, Scott Crosby sc...@sacrosby.com wrote: Over the last few months, several bugs have been reported where the pbf reader seems to sometimes not work correctly on Windows. The bug was caused by a 32-bit cleanness bug in the shared osmpbf library, where the reader

[OSM-dev] osmpbf v1.1.1 jar released. It should fix the PBF-on-windows issues.

2011-01-25 Thread Scott Crosby
Over the last few months, several bugs have been reported where the pbf reader seems to sometimes not work correctly on Windows. The bug was caused by a 32-bit cleanness bug in the shared osmpbf library, where the reader would erroneously sense the end of a file and cease reading. For whatever

Re: [OSM-dev] osmosis pbf file too small

2011-01-04 Thread Scott Crosby
There appears to be a 32-bit problem with java on Windows where the java runtime reports a spurious end-of-file indication to the reader, which then assumes the file has been completely read. This occurs even with 64-bit Java on 64-bit windows. In your particular case, it believes the file ends at

[osmosis-dev] PBF version 1.1, a PBF conformance file, and changes to Osmosis's PBF support

2011-01-02 Thread Scott Crosby
I am releasing version 1.1 and have just pushed several changes to osmosis trunk. These changes don't effect the format, but may require minor changes to software that uses the *.proto definitions. Although it has been in trunk for a while, this is the first release where the generated C/C++

[OSM-dev] PBF version 1.1, a PBF conformance file, and changes to Osmosis's PBF support

2011-01-02 Thread Scott Crosby
I am releasing version 1.1 and have just pushed several changes to osmosis trunk. These changes don't effect the format, but may require minor changes to software that uses the *.proto definitions. Although it has been in trunk for a while, this is the first release where the generated C/C++

Re: [OSM-dev] PBF HeaderBlock optional?

2010-12-21 Thread Scott Crosby
On Mon, Dec 20, 2010 at 6:01 PM, Scott Crosby sc...@sacrosby.com wrote: Yes. That is an osmosis bug. That header block is required. I would like to clarify this slightly. There must be an OSMHeader block before the first OSMData block, however, the OSMHeader block does not have to be the first

Re: [OSM-dev] PBF HeaderBlock optional?

2010-12-21 Thread Scott Crosby
On Tue, Dec 21, 2010 at 10:28 AM, Jochen Topf joc...@remote.org wrote: On Tue, Dec 21, 2010 at 10:08:41AM -0600, Scott Crosby wrote: On Mon, Dec 20, 2010 at 6:01 PM, Scott Crosby sc...@sacrosby.com wrote: Yes. That is an osmosis bug. That header block is required. I would like to clarify

Re: [OSM-dev] Merging pbf files problem?

2010-12-21 Thread Scott Crosby
Yeah, I know. I'm using the SVN sources, and compiled with protobuf-c. As I said, I was able to load individual pbf extracts no problem; it's just that osm2pgsql dies immediately on the combined .pbf extract. That's why I'm thinking that Osmosis' pbf output might be bugged. I'm happy to try

Re: [OSM-dev] PBF HeaderBlock optional?

2010-12-21 Thread Scott Crosby
On Tue, Dec 21, 2010 at 6:47 PM, Anthony o...@inbox.org wrote: On Tue, Dec 21, 2010 at 11:08 AM, Scott Crosby sc...@sacrosby.com wrote: On Mon, Dec 20, 2010 at 6:01 PM, Scott Crosby sc...@sacrosby.com wrote: Yes. That is an osmosis bug. That header block is required. I would like to clarify

Re: [OSM-dev] Merging pbf files problem?

2010-12-20 Thread Scott Crosby
Also, does .pbf format support changesets yet? No idea. Since most changesets are small the time and space savings of PBF are generally less important. Exactly. There is no changeset-PBF format at present for the reasons you stated. Scott ___

Re: [OSM-dev] Delta-encoding in PBF

2010-12-18 Thread Scott Crosby
On Sat, Dec 18, 2010 at 6:28 AM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 18-12-10 11:53, David Paleino schreef: Hello people, I'm writing a parser in C# for PBF dumps. However, I'm having issues with the Delta-{en|de}coding.

Re: [osmosis-dev] pbf problems under windows

2010-12-15 Thread Scott Crosby
On Wed, Dec 15, 2010 at 5:24 AM, Chris66 chris66...@gmx.de wrote: Am 15.12.2010 11:48, schrieb Chris66: Some more tests: NO effect (same sized output file) have: java u23 (32 Bit) -Xmx1024 setting -XX:-ReduceInitialCardMarks (gives 'unknown VM-option' error) osmosis V0.37 Interesting

Re: [osmosis-dev] pbf problems under windows

2010-12-14 Thread Scott Crosby
On Tue, Dec 14, 2010 at 2:10 AM, aighes h.scholl...@googlemail.com wrote: Hello, I proceed the Geofabrik files of 13.12. europe.osm.pbf (Geofabrik): * MD5: 9D278EE963925ED34701431B8573E0BF * 4448335290 bytes Just to confirm, this file from geofabrik is broken? europe.osm.bz2

Re: [OSM-dev] Storing way/relation start offset in PBF file

2010-12-14 Thread Scott Crosby
There's a way you could split it up a lot faster. There's no need to decompress each block. Just copy the blocks that contain entities that you're interested in. Just a data point for doing this: I implemented a quick Java program that CRC's every block in a PBF file. It decompresses each

Re: [osmosis-dev] pbf problems under windows

2010-12-14 Thread Scott Crosby
On Tue, Dec 14, 2010 at 10:33 AM, Frederik Ramm frede...@remote.org wrote: Hi,   now I have the files ready for your inspection, on http://www.geofabrik.de/tmp/ MD5 sums: 9d278ee963925ed34701431b8573e0bf  europe-created-on-linux.osm.pbf c4186d9197225e97f5bc935be09881e8  

Re: [osmosis-dev] pbf problems under windows

2010-12-13 Thread Scott Crosby
On Mon, Dec 13, 2010 at 2:12 AM, Frederik Ramm frede...@remote.org wrote: Hi,   I have a number of reports where Windows users complain that they cannot process my (osmosis-generated) .osm.pbf extracts under Windows. For some it seems to break with country extracts but it seems that everybody

[OSM-dev] Building the planet splitter from the crosby_integration branch

2010-12-04 Thread Scott Crosby
Just a quick note. This is the code for splitting the planet into thousands of areas in a single pass based on the mkgmap splitter. I mentioned this code on both osm-dev and mkgmap-dev. This branch supports PBF, runs much faster, threads better, doesn't have the 'node in too many areas' issue,

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-02 Thread Scott Crosby
On Thu, Dec 2, 2010 at 2:51 AM, Sarah Hoffmann lon...@denofr.de wrote: Hi Andrzej, On Wed, Dec 01, 2010 at 03:19:18PM +0100, andrzej zaborowski wrote: A further comment on splitting a big dataset into areas is that if the areas are disjoint (like in the case of countries, provinces and other

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-02 Thread Scott Crosby
On Thu, Dec 2, 2010 at 2:06 PM, Frederik Ramm frede...@remote.org wrote: I wanted to try this but it seems I have difficulties collecting the various dependencies. I got hold of the fastutils.jar but:    [javac] /home/fred/splitter/src/uk/me/parabola/splitter/DenseInt2ShortMap.java:5:

[OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-01 Thread Scott Crosby
I've been working on making various improvements to the mkgmap splitter. It was originally intended to split a map into smaller pieces which could then be converted into Garmin format and artifacts of that original purpose continue to exist. (Eg, its coordinate system for representing the areas.)

Re: [OSM-dev] Compression types in PBF Format

2010-12-01 Thread Scott Crosby
On Wed, Dec 1, 2010 at 9:50 AM, Anthony o...@inbox.org wrote: On Wed, Dec 1, 2010 at 10:47 AM, Stefan de Konink ste...@konink.de wrote: On Wed, 1 Dec 2010, Anthony wrote: Yeah, but your lead basically shows we are talking about more than 10%... Yeah, probably, but at the expense of more

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-01 Thread Scott Crosby
On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets nro...@gmail.com wrote: Hello Scott, How do you keep track of what bboxs each entity belongs to ? An Int2ShortMultiMap implemented by composing two underlying Int2ShortMap implementations with different space efficiency tradeoffs, a custom sparsearray

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-01 Thread Scott Crosby
On Wed, Dec 1, 2010 at 12:33 PM, Nic Roets nro...@gmail.com wrote: On Wed, Dec 1, 2010 at 8:18 PM, Scott Crosby scro...@cs.rice.edu wrote: On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets nro...@gmail.com wrote: Hello Scott, How do you keep track of what bboxs each entity belongs

Re: [OSM-dev] Block sizes in PBF Format

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 3:28 AM, Jochen Topf joc...@remote.org wrote: The PBF_Format wiki page states: The length of a Blob *should* be less than 16 megabytes and *must* be less than 32 megabytes. But forther down it says I collect 8k entities to form a PrimitiveBlock, which is serialized

Re: [OSM-dev] Compression types in PBF Format

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 2:21 AM, Jochen Topf joc...@remote.org wrote: The PBF format supports three compression types: zlib, lzma, and bzip2. Do we have to support all of them? What is the currently existing software using? Good question. I think that the bzip2 compression option is useless.

Re: [OSM-dev] Storing way/relation start offset in PBF file

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 2:26 PM, Jochen Topf joc...@remote.org wrote: Sometimes it is useful to read ways before nodes or relations before way or nodes. With the XML format this is not really possible, but with the PBF format it could be reasonably easy if we store the offsets in the file where

Re: [OSM-dev] Compression types in PBF Format

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 2:41 PM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 ...so all code of Jochen should be used now? Get real. So exactly what Scott suggest: why does nobody step in then, write code that nobody uses afterwards and present a

Re: [OSM-dev] Compression types in PBF Format

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 7:29 PM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Scott, Op 01-12-10 00:41, Scott Crosby schreef: The real question is does supporting bzip2/lzma offer advantages that are commensurate with the added implementation

Re: [OSM-dev] Storing way/relation start offset in PBF file

2010-11-30 Thread Scott Crosby
On Tue, Nov 30, 2010 at 5:47 PM, Jochen Topf joc...@remote.org wrote: On Tue, Nov 30, 2010 at 05:07:26PM -0600, Scott Crosby wrote: On Tue, Nov 30, 2010 at 2:26 PM, Jochen Topf joc...@remote.org wrote: One question however. Why? If you're doing this a lot, why isn't it acceptable to have 3

Re: [OSM-dev] pbf2osm bug: way and relation timestamps in 1969

2010-11-22 Thread Scott Crosby
On Mon, Nov 22, 2010 at 10:14 AM, Frederik Ramm frede...@remote.org wrote: Dear pbf2osm devs, please consider this example: This file represents relation #882 as downloaded from the API with relation/882/full: http://www.remote.org/frederik/tmp/r882orig.osm I convert this to pbf using

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-11-18 Thread Scott Crosby
On Thu, Nov 18, 2010 at 4:47 PM, Lennard l...@xs4all.nl wrote: On 16-10-2010 9:55, Lennard wrote: On 16-10-2010 8:39, Brett Henderson wrote: If anybody sees any issues with the binary support, please let Scott and I know. I'm now building Osmosis via an automated Hudson process so pushing

Re: [OSM-dev] PBF: add a package keyword to the .proto for CPP namespace

2010-11-16 Thread Scott Crosby
On Tue, Nov 16, 2010 at 10:52 AM, Chris Browet c...@semperpax.com wrote: Hi, Would it be possible to add a package zzz; to the .proto files, as in google examples? In CPP, that will generate a namespace named zzz, which would avoid class name clashing (mind you, I happen to already have

Re: [OSM-dev] Planet in PBF?

2010-11-15 Thread Scott Crosby
On Mon, Nov 15, 2010 at 5:50 AM, Emilie Laffray emilie.laff...@gmail.comwrote: If we were to drop the osm.bz2 file, we will give ample warnings and long in advance. In addition, I strongly believe that the XML OSM file format is a good exchange format so I don't see it disappearing. The usage

Re: [OSM-dev] Planet in PBF?

2010-11-14 Thread Scott Crosby
On Sun, Nov 14, 2010 at 4:54 AM, Stefan de Konink ste...@konink.de wrote: Is it possible to generate them and be placed on planet.openstreetmap.org? I'm guessing that it could be done, but you need to ask whoever runs that server to do it. Scott ___

Re: [OSM-dev] Planet in PBF?

2010-11-13 Thread Scott Crosby
On Thu, Nov 11, 2010 at 4:06 AM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, Did anyone already made an attempt to generate a (full) Planet PBF file? Yes. I have made several full planets when doing my size and speed benchmarks. Scott

Re: [osmosis-dev] piping pbf

2010-11-09 Thread Scott Crosby
No reason, except that I hadn't written the command line parser to map '-' to System.in Scott ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] pbf2osm development has started [code to test!]

2010-10-25 Thread Scott Crosby
On Mon, Oct 25, 2010 at 4:15 AM, Stefan de Konink ste...@konink.de wrote: On Mon, 25 Oct 2010, Chris Browet wrote: Sure. I don't think the C - C++ will be problematic. I have been thinking about 'hooks' so the defines can become hooks you replace (So somesort of ifndef way.) Or just

Re: [OSM-dev] Binary Format, Performance, osm2pbf

2010-10-17 Thread Scott Crosby
On Fri, Oct 15, 2010 at 8:56 AM, Frederik Ramm frede...@remote.org wrote: Hi, a few weeks ago when I announced that I am planning to rely on the new binary format in the future, I caught some flak for claiming that an .osm.pbf was not only faster to produce, parse, and transmit than a .bz2

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 6:36 AM, Frederik Ramm frede...@remote.org wrote: Scott, Thanks, I enjoyed looking at your code and style of coding. The code is Stefan's. I only wrote about an idea I had to wrap it up into something more convenient - but I wasn't involved until now. I think

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 12:42 PM, Frederik Ramm frede...@remote.org wrote: Hi, Scott Crosby wrote: I think the perfect permanent place for Stefan's implementation, and your idea of wrapping it up as a debian package, is with the http://github.com/scrosby/OSM-binary main repository

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
Not what I want, because (1) every project will have to have a maintainer who decideds what gets pulled into trunk - this introduces more formality than we have now; (2) as a committer, I want it to be *my* decision whether I commit something right away - in cases where I'm sure it is good

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 2:40 PM, Greg Troxel g...@ir.bbn.com wrote: I think what's going on now is that we have a svn repo that git weenies don't particularly care for, and no git repo support on osm servers, and that people are therefore basically setting up git repos elsewhere for various

Re: [OSM-dev] Binary Format, Performance, pbf2osm

2010-10-17 Thread Scott Crosby
On Sun, Oct 17, 2010 at 6:34 AM, Stefan de Konink ste...@konink.de wrote: On Sun, 17 Oct 2010, Stefan de Konink wrote: message 'PrimitiveBlock': missing required field 'stringtable' Error unpacking PrimitiveBlock message (gdb) print *hmsg $3 = {base = {descriptor = 0x406740,

Re: [OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-16 Thread Scott Crosby
On Sat, Oct 16, 2010 at 1:39 AM, Brett Henderson br...@bretth.com wrote: I've released version 0.37 of Osmosis with this included. I haven't had a chance to update the wiki either. I'll try to do so over the next few days if nobody beats me to it. I just updated the Wiki. Scott

[OSM-dev] OSM binary format (pbf) 1.0 is in osmosis trunk.

2010-10-12 Thread Scott Crosby
On Thu, Sep 16, 2010 at 10:23 AM, Scott Crosby scro...@cs.rice.edu wrote: For now, for simplicity, I'm going to revert to the same metadata as the XML format. Just a BBox and source field. I'll make them both optional, making it easier to upgrade metadata features in the future. When

Re: [OSM-dev] Bin-test failed (different test)

2010-10-03 Thread Scott Crosby
On Thu, Sep 30, 2010 at 7:41 PM, Brett Henderson br...@bretth.com wrote: On Fri, Oct 1, 2010 at 12:23 AM, Curt Nowak no...@bwl.uni-hildesheim.dewrote: Hello dev, I also tested osmosis 0.37-SNAPSHOT with the bin-support for various countries downloaded at Geofabrik. In contrast to the

Re: [OSM-dev] pbf2osm development has started [code to test!]

2010-09-30 Thread Scott Crosby
On Thu, Sep 30, 2010 at 4:44 AM, Stefan de Konink ste...@konink.de wrote: - allow the use of the bbox What exactly do you mean by this: ? Filter output by applying a bbox (the expensive way). Extract nodes within a bbox, sort them on ID, extract all ways, validate per item if

Re: [OSM-dev] pbf2osm development has started [code to test!]

2010-09-30 Thread Scott Crosby
On Thu, Sep 30, 2010 at 6:04 AM, Stefan de Konink ste...@konink.de wrote: On Thu, 30 Sep 2010, Scott Crosby wrote: I proposed exactly this for the mkgmap splitter. If you're going to do this, can I propose a tweak where you can output thousands of files simultaneously? The differences

Re: [OSM-dev] Bin-test failed

2010-09-30 Thread Scott Crosby
On Thu, Sep 30, 2010 at 11:09 AM, Christian H. Bruhn br...@arcor.de wrote: Hello Scott, Tuesday, September 28, 2010, 9:09:16 PM, you wrote: Is your problem repeatable? Yes. If it is repeatable, one explanation is that a 'too big' block was generated, and somehow the warning was

Re: [OSM-dev] pbf2osm development has started [code to test!]

2010-09-29 Thread Scott Crosby
But what I already discussed with Scott, we *need* a good 'has everything' PBF file. Something that can test a parser and has expected output. I know. I know. I need to hack a custom program that generates the entire protocol buffer, serializes it, hitting all of the interesting features and

Re: [OSM-dev] Bin-test failed

2010-09-28 Thread Scott Crosby
On Sat, Sep 25, 2010 at 2:51 AM, Christian H. Bruhn br...@arcor.de wrote: Friday, September 24, 2010, 6:06:59 PM, Scott Crosby wrote: Is there an error message thrown? If so, please paste it. No error message, everything seems to be OK. If the error message is something about a possibly

Re: [OSM-dev] Bin-test failed

2010-09-26 Thread Scott Crosby
On Sat, Sep 25, 2010 at 2:51 AM, Christian H. Bruhn br...@arcor.de wrote: Friday, September 24, 2010, 6:06:59 PM, Scott Crosby wrote: Is there an error message thrown? If so, please paste it. No error message, everything seems to be OK. If the error message is something about a possibly

Re: [OSM-dev] Bin-test failed

2010-09-24 Thread Scott Crosby
On Fri, Sep 24, 2010 at 2:11 AM, Christian H. Bruhn br...@arcor.de wrote: Hello Peter, Thursday, September 23, 2010, 10:07:03 PM, you wrote: Am 23.09.2010 19:35, schrieb Christian H. Bruhn: seems to work properly (for only 60 seconds) but only produces a pbf-file with 128 MB? Does it

Re: [OSM-dev] pbf2osm development has started

2010-09-23 Thread Scott Crosby
On Thu, Sep 23, 2010 at 9:01 PM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 03:44, Stefan de Konink schreef: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Never mind,

Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-22 Thread Scott Crosby
On Wed, Sep 22, 2010 at 2:17 AM, Frederik Ramm frede...@remote.org wrote: Hi, Kai Krueger wrote: Easy to do - just download the .osm.pbf and run osmosis --read-bin country.osm.pbf --write-xml country.osm May I suggest it be run on planet.openstreetmap.org then as part of the core

Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-21 Thread Scott Crosby
On Tue, Sep 21, 2010 at 5:30 AM, Frederik Ramm frede...@remote.org wrote: Hi, Scott Crosby wrote: This has been done. RC2 is in osmosis trunk. Changes are almost exclusively to the underlying osmbin.jar with no format incompatibilities. Changes include: I've been using it for a few

Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-12 Thread Scott Crosby
On Sat, Sep 4, 2010 at 10:51 PM, Scott Crosby scro...@cs.rice.edu wrote: The OSM binary interchange fileformat is now part of osmosis trunk and is fully functional. This is the last chance to make potentially incompatible changes. In a week or two, I will add the design doc and command line

Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-05 Thread Scott Crosby
On Sun, Sep 5, 2010 at 7:02 AM, Peter Körner osm-li...@mazdermind.de wrote: Am 05.09.2010 05:51, schrieb Scott Crosby: This schema is more expressive than the current XML which only includes a 'source' field in thebounds  tag. Should I alter my schema to have just one field, 'source

Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-05 Thread Scott Crosby
On Sun, Sep 5, 2010 at 11:42 AM, Frederik Ramm frede...@remote.org wrote: Scott, Scott Crosby wrote: message HeaderBlock {  required HeaderBBox bbox = 1;  // Author, name, and version number of the dataset in this file. (to permit  // patches/updates to be incrementally applied

[OSM-dev] Release candidate for OSM binary format is in osmosis trunk.

2010-09-04 Thread Scott Crosby
The OSM binary interchange fileformat is now part of osmosis trunk and is fully functional. This is the last chance to make potentially incompatible changes. In a week or two, I will add the design doc and command line options doc to the wiki. After that, schema changes will be much more

Re: [OSM-dev] Reading OSM History dumps

2010-08-23 Thread Scott Crosby
On Mon, Aug 23, 2010 at 8:39 AM, Frederik Ramm frede...@remote.org wrote: Hi, Scott Crosby wrote: My idea is that we assume that a given snapshot of the database at a given changeset number is unique. I don't know if that breaks your whole edifice but the above assumption is false

Re: [OSM-dev] Reading OSM History dumps

2010-08-23 Thread Scott Crosby
On Mon, Aug 23, 2010 at 9:48 AM, Lars Francke lars.fran...@gmail.com wrote: Creating a true history is impossible. The information about the concurrent updates across different non-atomic changesets is presumably lost. You can easily recreate that by sorting everything by timestamp. Which I

Re: [OSM-dev] Proposed schema for the OSM binary format.

2010-08-10 Thread Scott Crosby
On Tue, Aug 10, 2010 at 12:33 AM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: hi scott, what has changed since the beginnning? See my other email. The core is the same, but some stuff has changed on the edges. the only thing that I would like to see is some form of

Re: [OSM-dev] New OSM binary fileformat implementation.

2010-08-09 Thread Scott Crosby
On Wed, Apr 28, 2010 at 12:02 PM, Scott Crosby scrosb...@gmail.com wrote: Hello! I would like to announce code implementing a binary OSM format that supports the full semantics of the OSM XML. It is 5x-10x faster at reading and writing and 30-50% smaller; an entire planet, including all

Re: [OSM-dev] New OSM binary fileformat implementation.

2010-08-09 Thread Scott Crosby
On Wed, Aug 4, 2010 at 7:17 PM, Brett Henderson br...@bretth.com wrote: On Tue, Aug 3, 2010 at 11:37 PM, Scott Crosby scro...@cs.rice.edu wrote: On Sun, Aug 1, 2010 at 6:39 AM, Brett Henderson br...@bretth.com wrote: If we go down this path I need two things: 1. A versioned jar file

[OSM-dev] Proposed schema for the OSM binary format.

2010-08-09 Thread Scott Crosby
As I'm in the process of packaging the osm binary format for packaging as part of osmosis, I expect that it may start get used fairly soon. Like any fileformat, the binary format requires a schema which will be hard to change. Before finalizing it, I would like to run my schema design past

Re: [OSM-dev] New OSM binary fileformat implementation.

2010-08-09 Thread Scott Crosby
On Sun, Aug 1, 2010 at 5:51 AM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: Hi, ist there any documentation of the binary format changes? I have implemented a c++ reader using protobuf, would update that if there is a new format spec. mike No real docs. There are some

Re: [OSM-dev] New OSM binary fileformat implementation.

2010-08-03 Thread Scott Crosby
On Sat, Jul 31, 2010 at 11:26 AM, Frederik Ramm frede...@remote.org wrote: Scott, others, Scott Crosby wrote: I would like to announce code implementing a binary OSM format that supports the full semantics of the OSM XML. [...] The changes to osmosis are just some new tasks to handle

Re: [OSM-dev] New OSM binary fileformat implementation.

2010-08-03 Thread Scott Crosby
On Sun, Aug 1, 2010 at 6:39 AM, Brett Henderson br...@bretth.com wrote: I'll help incorporate this into the rest of Osmosis.  There's a few things to work through though. I don't have a lot of time to work with this, but I can split up my working branch (which includes several unrelated

Re: [OSM-dev] Using relations to save data

2010-05-04 Thread Scott Crosby
On Tue, May 4, 2010 at 3:12 PM, Andreas Kalsch andreaskal...@gmx.de wrote: I am still reading some old mailing list posts ... What about a relation with type=data, which is a relation that can include tags and other relations recusively? This relation has no geometric reference but it is

  1   2   >