Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

On Wed, Dec 1, 2010 at 12:33 PM, Nic Roets wrote: > > > On Wed, Dec 1, 2010 at 8:18 PM, Scott Crosby wrote: >> >> On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets wrote: >> > Hello Scott, >> > >> > How do you keep track of what bboxs each entity belongs to ? >> >> An Int2ShortMultiMap implemented by co

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 12:40 PM, Scott Crosby wrote: > The 15% gain you measured between .rawpbf.xz and .pbf  really lets > lzma cheat too much, because it can exploit a window tens of times > larger than it would if integrated. I'm not sure how much that mattered. xz -3, which I believe uses a

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

On Wed, Dec 1, 2010 at 8:18 PM, Scott Crosby wrote: > On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets wrote: > > Hello Scott, > > > > How do you keep track of what bboxs each entity belongs to ? > > An Int2ShortMultiMap implemented by composing two underlying > Int2ShortMap implementations with differ

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets wrote: > Hello Scott, > > How do you keep track of what bboxs each entity belongs to ? An Int2ShortMultiMap implemented by composing two underlying Int2ShortMap implementations with different space efficiency tradeoffs, a custom sparsearray implementation

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 9:50 AM, Anthony wrote: > On Wed, Dec 1, 2010 at 10:47 AM, Stefan de Konink wrote: >> On Wed, 1 Dec 2010, Anthony wrote: >> Yeah, but your lead basically shows we are talking about more than 10%... >>> >>> Yeah, probably, but at the expense of more complicated code, gr

Re: [OSM-dev] Compression types in PBF Format

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 01-12-10 17:30, Anthony schreef: > Anyway, I'm probably completely wrong about this. Sorry. I guess the fastest way to verify all this is someone that adds the LZMA and BZ2 library to java and check in osmosis. Your numbers give me the impressio

Re: [OSM-dev] Storing way/relation start offset in PBF file

On Tue, Nov 30, 2010 at 09:48:37PM -0600, Scott Crosby wrote: > > It has > > more to do with the file serialization than with the file contents.  Its > > just > > something that concerns the tasks reading and writing the pbf.  It has to > > remember those offsets and add them to the end of the fil

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 10:55 AM, Stefan de Konink wrote: > On Wed, 1 Dec 2010, Anthony wrote: > >> Not in an embedded system, which is where a small difference like 10% >> is going to matter. > > Please elaborate? Either the memory is used for a block cache or for the > program. No idea. If you

Re: [Potlatch-dev] [OpenStreetMap] #3357: Drop-down autocomplete for 'bulding' lists "yes" multiple times

#3357: Drop-down autocomplete for 'bulding' lists "yes" multiple times +--- Reporter: dandv | Owner: potlatch-...@… Type: defect | Status: closed Priorit

Re: [OSM-dev] Compression types in PBF Format

On Wed, 1 Dec 2010, Anthony wrote: Not in an embedded system, which is where a small difference like 10% is going to matter. Please elaborate? Either the memory is used for a block cache or for the program. I'm interested now in seeing how the full history compression goes, though.  If it

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 10:47 AM, Stefan de Konink wrote: > On Wed, 1 Dec 2010, Anthony wrote: > >>> Yeah, but your lead basically shows we are talking about more than 10%... >> >> Yeah, probably, but at the expense of more complicated code, greater >> memory usage, etc. > > The hole process is IO-

Re: [OSM-dev] Compression types in PBF Format

On Wed, 1 Dec 2010, Anthony wrote: Yeah, but your lead basically shows we are talking about more than 10%... Yeah, probably, but at the expense of more complicated code, greater memory usage, etc. The hole process is IO-bound... memory is used anyway to overcome the IO issues... I'm inte

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 10:24 AM, Stefan de Konink wrote: > On Wed, 1 Dec 2010, Anthony wrote: > >>> Did you benchmark what pbf + lzma did or did you embed lzma in osmosis? >> >> xz uses lzma.  I made an uncompressed pbf file (florida.osm.rawpbf) >> and then compressed it with xz (florida.osm.rawpb

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 10:35 AM, Peter Körner wrote: > Am 30.11.2010 23:44, schrieb Anthony: >> >> On Tue, Nov 30, 2010 at 5:19 PM, Matt Amos  wrote: >>> because XML is a nearly human-readable, easy to explain and inspect >>> format. >> >> Except when you don't include any line feeds :). > > What

Re: [OSM-dev] Compression types in PBF Format

Am 30.11.2010 23:44, schrieb Anthony: On Tue, Nov 30, 2010 at 5:19 PM, Matt Amos wrote: On Tue, Nov 30, 2010 at 8:41 PM, Stefan de Konink wrote: And if we can change the API every week, I wonder why we are still at XML then. because XML is a nearly human-readable, easy to explain and inspec

Re: [OSM-dev] Compression types in PBF Format

On Wed, 1 Dec 2010, Anthony wrote: Did you benchmark what pbf + lzma did or did you embed lzma in osmosis? xz uses lzma. I made an uncompressed pbf file (florida.osm.rawpbf) and then compressed it with xz (florida.osm.rawpbf.xz). This isn't the same as making a pbf file which uses lzma, but

Re: [OSM-dev] Compression types in PBF Format

On Wed, Dec 1, 2010 at 9:28 AM, Stefan de Konink wrote: > On Wed, 1 Dec 2010, Anthony wrote: >> LZMA vs. zlib actually makes less of a difference than I thought it would: >> >> -rw-r--r-- 1 a a 103M 2010-12-01 08:07 florida.osm.bz2 >> -rw-r--r-- 1 a a 129M 2010-12-01 08:32 florida.osm.gz >> -rw-r-

Re: [OSM-dev] Bing in JOSM

Am 01.12.2010 12:20, schrieb Peter Körner: I created a tms2bing redirector Just forget about it, I didn't see Ians solution and this was my shortest road to success. Thank you Ian for your clean solution: Peter _

Re: [OSM-dev] Compression types in PBF Format

On Wed, 1 Dec 2010, Anthony wrote: On Tue, Nov 30, 2010 at 11:03 PM, Anthony wrote: On Tue, Nov 30, 2010 at 8:29 PM, Stefan de Konink wrote: If any of gzip/bzip2/lzma in the general give better compression ratio's (20% smaller), then this compression scheme should become the default format.

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

Hi Nic and Scott, On 1 December 2010 14:27, Nic Roets wrote: > http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp?rev=24484 A further comment on splitting a big dataset into areas is that if the areas are disjoint (like in the case of countries, provinces and othe

Re: [OSM-dev] Compression types in PBF Format

On Tue, Nov 30, 2010 at 11:03 PM, Anthony wrote: > On Tue, Nov 30, 2010 at 8:29 PM, Stefan de Konink wrote: >> If any of gzip/bzip2/lzma in the general give better compression ratio's >> (20% smaller), then this compression scheme should become the default >> format. > > Depends on the performanc

Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

Hello Scott, How do you keep track of what bboxs each entity belongs to ? I'm not really asking a question, I'm just saying that I found a way to reduce the memory requirement for that considerably. Instead of a bit per bbox per entry, I store only 16 bits or 32 bits per entry. Here is the source

[OSM-dev] Splitting the planet into thousands of pieces in one pass.

I've been working on making various improvements to the mkgmap splitter. It was originally intended to split a map into smaller pieces which could then be converted into Garmin format and artifacts of that original purpose continue to exist. (Eg, its coordinate system for representing the areas.)

[OSM-dev] Bing in JOSM

Hi I created a tms2bing redirector that runs on php & apache. It can be used together with the slippymap plugin in josm. I tested it on localhost but I'm hesitating to host it on my private server as long as nobody checked if it is okay to use the bing api like that. Especially the usage not