For Merkaartor, I also implemented a Binary file format (see http://wiki.openstreetmap.org/index.php/Osb_spec_v2)
Key points are: - Non lossy (it is editor oriented, rather than routing or render) - avoid tags Key/values duplication (there are 2 tables with unique string, features only holds pointers) - Regions + tile based for memory efficiency There is nowadays no provision fur updates/diff, although the region-based paradigm allows specific updates. Note also that, internally, Merkaartor uses signed int32. Unless I'm wrong, 40.000 km / 2^32 = 10mm, which seems good enough with GPSes with a 10m precision ;-) Just my 2 cents. - Chris - 2008/11/9 Stefan de Konink <[EMAIL PROTECTED]> > Marcus Wolschon wrote: > > Stefan de Konink schrieb: > >> The two problems I see; - In order to allow updates there should be > >> some form of 'updatable space'. - If this space is not present it > >> might be good to have one file that contains *all* strings, another > >> one that contains the rest of the data, and maybe a final > >> client-side generated index on both of them. > > I do this with large-strings(Strings with <32 characters are inline) > > and separate files for nodes, ways, indexes, ... > > I thought about storing all strings externally but ended up noting > > that most tags are not very long. > > The point is not that tags are long but that all keys are duplicates :) > > > My purpose is not minimal bandwith for transmission but a good on-disk > > - -format, so my metrics are different > > from yours. > > True. Don't forget that my first pass encoding, is like you want it to > be. Maybe you could skip 'users' and 'timestamps' from the data. This > would significantly reduce the amount of data. > > > No. Attributes that are longer then the 32 bytes are stored in an > > external file. > > Nearly all attributes fit in here, so most accesses require no > > additional seek. > > Ok :) Sounds like Paradox :) :) :) Good! > > >>> I am trying something similar but with fixed length records and > >>> back-links from node to way to allow updates to be applied to the > >>> file. 1.4GB-135MB is nice but you still don't want to download > >>> 135MB every day to have an up-to-date netherlands-file (let alone > >>> to do this for the planet). > >> 135MB (gzip XML) -> 78MB (bzip2 bin) > >> > >> We are just looking at the possibilities to binary diff the files, > >> just to allow partial updates. By XORing them on the source present > >> at the user. > > > > Interesting. It could make a good download-format. > > I'm looking forward to seeing this happen. :) > > How do you intend to handle the boundary-rectangles for diffs > > if a user does not store all the world on e.g. his small nettop? > > Use binary-files and diffs per country? > > Binary files per country sounds the most reasonable thing to do. The > other problem that the file only will grow bigger, and get fragmentation > problems, is something else. So we might have to implement a search and > reorder every 3 months. > > > >>> I was quite occupied with another open-source-project of mine and > >>> switching jobs but now I should have the time to finish > >>> implementing my own proposal in code and test it's performance. > >> :) good luck :) if you want to team up to write the ultimate code, > >> just send a private mail. > > I just started implementing the memory-mapped io-code for my nodes.obm. > > First I want to get this brainchild of mine going and then implement > > a Java-parser > > for your format for binary-downloads. > > This could get really fast. :) > > At #osm-nl we ar discussing the float -> long thing. I used floats > because it (obviously) allows more precision, but I agree on some points > mentioned before. > > > Stefan > > _______________________________________________ > dev mailing list > [email protected] > http://lists.openstreetmap.org/listinfo/dev >
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
