Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
Any idea when a final version of the file format will be available? I plan to provide an importer plugin for the binary format in MoNav pretty soon, as reading the XML file takes up most of the preprocessing phase right now. Greetings Christian Vetter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Centroid-Column - Patch for osm3pgsql
Hi
A while ago we talked about adding support for a centroid column to
osm2pgsql [1]. Attached you'll find a patch that adds this
functionality. I did not test it extensively, especially not with
invalid geometries, but for most cases it should work.
Unfortunately the server I tested on has only geos v3.1 and to use
interpolate on linestrings at least geos 3.2 ist required, so the
centroid point does not sit in the middle of the linesting but ..
somewhere else on the line.
Please, help me with testing and reviewing this patch.
Peter
[1]
http://lists.openstreetmap.org/pipermail/dev/2010-September/020536.html
Index: output.h
===
--- output.h(revision 23282)
+++ output.h(working copy)
@@ -29,7 +29,8 @@
const char *expire_tiles_filename; /* File name to output expired tiles
list to */
int enable_hstore; /* add an additional hstore column with objects key/value
pairs */
int enable_multi; /* Output multi-geometries intead of several simple
geometries */
- char** hstore_columns; /* list of columns that should be written into their
own hstore column */
+ int enable_centroid; /* Generate an additional geometry column that contains
the center of the geometry */
+ const char** hstore_columns; /* list of columns that should be written into
their own hstore column */
int n_hstore_columns; /* number of hstore columns */
};
Index: build_geometry.cpp
===
--- build_geometry.cpp (revision 23282)
+++ build_geometry.cpp (working copy)
@@ -21,6 +21,7 @@
*/
#include iostream
+#include cstdio
#include cstring
#include cstdlib
@@ -59,6 +60,7 @@
typedef std::auto_ptrGeometry geom_ptr;
static std::vectorstd::string wkts;
+static std::vectorstd::string wkt_centroids;
static std::vectordouble areas;
@@ -80,6 +82,9 @@
std::auto_ptrLinearRing
shell(gf.createLinearRing(coords.release()));
geom = geom_ptr(gf.createPolygon(shell.release(), new
std::vectorGeometry *));
geom-normalize(); // Fix direction of ring
+
+// this function seems not to be used anymore so centeroid
functionality is not implemented here
+// see get_wkt_split
} else {
if (coords-getSize() 2)
return NULL;
@@ -103,13 +108,13 @@
}
}
-
-size_t get_wkt_split(osmNode *nodes, int count, int polygon, double split_at) {
+size_t get_wkt_split(osmNode *nodes, int count, int polygon, double split_at,
int enable_centeroid) {
GeometryFactory gf;
std::auto_ptrCoordinateSequence
coords(gf.getCoordinateSequenceFactory()-create(0, 2));
double area;
WKTWriter wktw;
size_t wkt_size = 0;
+char centroid[100];
try
{
@@ -129,6 +134,15 @@
std::string wkt = wktw.write(geom.get());
wkts.push_back(wkt);
areas.push_back(area);
+
+// centroid of a closed way
+if(enable_centeroid)
+{
+Point* p = geom-getInteriorPoint();
+snprintf(centroid, sizeof(centroid), POINT(%.15g %.15g),
p-getX(), p-getY());
+wkt_centroids.push_back(centroid);
+}
+
wkt_size++;
} else {
if (coords-getSize() 2)
@@ -146,6 +160,23 @@
std::string wkt = wktw.write(geom.get());
wkts.push_back(wkt);
areas.push_back(0);
+
+// centroid of a linear way
+if(enable_centeroid)
+{
+// the interior point is always on an edge of the line
+Point* p = geom-getInteriorPoint();
+
+// the centroid is not always part of the line
+// Point* p = geom-getCentroid();
+
+// interpolate a point on the geometry, needs geos =
3.2
+// Point* p = geom-interpolate(0.5);
+
+snprintf(centroid, sizeof(centroid), POINT(%.15g
%.15g), p-getX(), p-getY());
+wkt_centroids.push_back(centroid);
+}
+
wkt_size++;
distance=0;
segment =
std::auto_ptrCoordinateSequence(gf.getCoordinateSequenceFactory()-create(0,
2));
@@ -181,6 +212,21 @@
return result;
}
+char * get_wkt_centroid(size_t index)
+{
+if( wkt_centroids.size() = index )
+return NULL;
+
+char *result;
+result = (char*) std::malloc( wkt_centroids[index].length() + 1);
+
+// At least give some idea of why we about to seg fault
+if (!result) std::cerr std::endl Unable to allocate memory:
[OSM-dev] Bin-test failed
Hi! I tested osmosis 0.37-SNAPSHOT with the bin-support. Why does the command osmosis.bat --read-xml-change c:\osm\Planet\20100909-20100910.osc.gz --read-bin c:\osm\Planet\planet.osm.pbf --apply-change --write-bin c:\osm\planet\planet2.osm.pbf seems to work properly (for only 60 seconds) but only produces a pbf-file with 128 MB? Christian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
On Wed, Sep 22, 2010 at 23:50, Frederik Ramm frede...@remote.org wrote: I always viewed the Geofabrik extracts as a service to the community who build interesting things, or try out stuff - and who can also be counted on for a little bit of willingness to adapt to change, or add a little extra program to their toolchain if required. You're making it sound as if the Geofabrik extracts are meanwhile a staple ingredient of consumer-facing services (just download gpsmid here and get your data from Geofabrik and you're good to go) - consumers who need to have things served on a silver platter and who cannot be bothered to install software or convert files or something. I'm not entirely comfortable with that. I think if you really are in a situation where you need a guaranteed 6 month notice period before anything changes, you should calculate some buffer on *your* side - e.g. you should definitely not assume that certain files will be at a certain place on a certain server all the time, but instead have some sort of directory that is under your control. You cannot burden Geofabrik - or any OSMer running something in his spare time - with keeping your application happy. Especially not if you only tell them about your application depending on them when they want to make a change! Having said that, I won't switch off the bz2 stuff tomorrow but I certainly think that I'd prefer to use Geofabrik resources to produce more extracts for more people than to keep things running smoothly for end users of an application somewhere. The Geofabrik download stuff is primarily intended to be low-level - a building block for members of our community, not a shrink-wrapped service for the world. I use the geofabrik services and I must say I'm slightly irked at the tone this discussion has taken. That you're providing these services for free *at all* is not a given, and it's great that you continue to do so. If you dropped osm.bz2 tomorrow with zero warning I'd at most have to download some file and add another osmosis command to some shellscripts. I can't see how that could possibly be a big deal for anyone, and hopefully the resources being freed up on your end by not grinding through gigabytes of XML every day could be spent on something more worthwhile as a result. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2shp reloaded
Hi, I've uploaded an all-new osm2shp to SVN (utils/export). It's done in C, it is quite fast, and versatile if you can write a few basic C statements (doesn't use a config file but code dealing with the shapefile set-up is neatly packed in its own source file). It can process large files (not tried on full planet but does all of Europe with ease). It is the same program I'm using to create the .shp.zip files on the download.geofabrik.de web site, and more or less a re-write of Jochen's Ruby osmlib export. It doesn't do relations however (and therefore no multipolygons either). I've taken the liberty of simply removing the old osm2shp from SVN (last modified in 2007) because I didn't think anyone was using it any more and I wanted the cool name for myself. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Bin-test failed
Am 23.09.2010 19:35, schrieb Christian H. Bruhn: seems to work properly (for only 60 seconds) but only produces a pbf-file with 128 MB? Does it work when you write to an xml file? Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Spatialite in OSM context
2010/9/22 Chris Browet c...@semperpax.com Hi, I wanted to know if someone already used Spatialite in an OSM context? For large datasets of OSM data, it seems to me like a good compromise between the heaviness of Posgresql and the verbosity (and memory consumption, as everything has to be loaded in memory) of XML. Thanks in advance for you feedback (if applicable). - Chris - ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev Hi, I react late online, as I spoke with Chris off-line. I am on the way to developpe such a converter named libosm (this one is mine, I known some others also exist). The very first time I need one week to import a Planet.osm in a 160 GB database SQLite3 / Spatialite. Now daily, I import the diff (in 2 ou 3 hours - i5 2.3 GHz - 4 GB ram - desktop PC). C / C++ Source code available freely at github/marcussacapuces91/ libosm. Still in development, not my pro business. -- Marc Sibert m...@sibert.fr ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
On Thu, Sep 23, 2010 at 1:49 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: hopefully the resources being freed up on your end by not grinding through gigabytes of XML every day could be spent on something more worthwhile as a result. Definitely. But how are the pbf files going to be created without grinding through gigabytes of XML? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
Hi, Ævar Arnfjörð Bjarmason wrote: I use the geofabrik services and I must say I'm slightly irked at the tone this discussion has taken. That you're providing these services for free *at all* is not a given, and it's great that you continue to do so. Thank you for saying that. - Then again there will probably be lots of people who don't know (or care) about the History of Everything and who rely on Geofabrik stuff being part of the system so I guess it was ok for the matter to be brought up - better have Kai tell me now than to get 20 angry emails a day once I really drop the .bz2s ;) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
Hi, Anthony wrote: Definitely. But how are the pbf files going to be created without grinding through gigabytes of XML? Well once you have a pbf planet, you only have to grind through the .osc files each day (--read-change diff.osc --read-bin planet.pbf --ac --write-bin new-planet.pbf). This is something that has now become a lot easier because before you would either have to reserve ~ 320 GB of HD space for that or pay a very hefty performance price by having Osmosis read from, and write to, compressed files. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Spatialite in OSM context
That work flow (osm2pgsql+PostGIS+pgsql2sqlite) does not help the end user very much. But perhaps someday there will be ready made Spatialite database files for download, with indexes and perhaps a bunch of views. Then it would be possible to download one single data file bundled with, let's say, QGis or OpenJUMP project file, or Mapnik style file or Mapserver mapfile, and then just run the application. I believe I will start building Spatialite and Mapserver mapfile bundles for Finland later this year. I've been working on such a thing a year ago - I managed to get Kosmos to save OSM data from an OSM XML file into a Spatialite database and then use it for rendering. The source code is currently in hibernation, since I switched my development efforts from Kosmos to Maperitive, but I'm planning to include it into Maperitive in the following months. Igor ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 As annouced some days ago I would start with the development if nobody stepped up before Friday. Nobody did, so I will do so. Language of constraint is C, we will most likely not use any external dependancies. So porting to different platforms including Windows should be trivial. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkybzzcACgkQYH1+F2Rqwn2a7ACeJsF2W+acO267enaHSTqkZMGD qC8AnAtLNbcwujGnt78CefxZH2nrUSiI =lmqw -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
Stefan, Stefan de Konink wrote: As annouced some days ago I would start with the development if nobody stepped up before Friday. Nobody did, so I will do so. I'm looking forward to it. - Any chance you'll be able to use Mike's work at http://github.com/h4ck3rm1k3/OSM-Osmosis? May give you a head start. Language of constraint is C, we will most likely not use any external dependancies. Well you won't get around the Google Protobuf stuff of course! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Spatialite in OSM context
All of this seems great. Now, if we could agree on a common DB schema, it seems to me that would make a great way to have large, usable (as in ready to use with zero install/config), portable (as in just put it on a usb key/cd/dvd) OSM datasets. - Chris - ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
On Thu, Sep 23, 2010 at 6:18 PM, Frederik Ramm frede...@remote.org wrote: Language of constraint is C, we will most likely not use any external dependancies. Well you won't get around the Google Protobuf stuff of course! The Google Protobuf stuff is in C++. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 02:03, Anthony schreef: On Thu, Sep 23, 2010 at 6:18 PM, Frederik Ramm frede...@remote.org wrote: Language of constraint is C, we will most likely not use any external dependancies. Well you won't get around the Google Protobuf stuff of course! The Google Protobuf stuff is in C++. There is a nice C variant of it too. http://code.google.com/p/protobuf-c/ But I find things like the following more interesting: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Since I don't get a decompression error, I wonder if osmosis is using a different proto than I am doing ;) Questions ;) Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycAooACgkQYH1+F2Rqwn2I7gCfd2WmfCRjKApOvTX3SXOOoYr+ aKIAmwZNyr2a0nWy32F5j+O6kz5M3e1t =zs57 -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 03:44, Stefan de Konink schreef: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Never mind, I expected protobufs to be more robust ;) It isn't. Fixed. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycBoIACgkQYH1+F2Rqwn08HQCggUlcXRfJxCL4ayfRrie+Bepp HWgAni/WQrEoBngiE0TcPWqEp1WCmd9e =Ce/d -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
On Thu, Sep 23, 2010 at 9:01 PM, Stefan de Konink ste...@konink.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 03:44, Stefan de Konink schreef: unsupported tag 4 at offset 16 Error unpacking compressed HeaderBlock message (Generated with osmosis...) Never mind, I expected protobufs to be more robust ;) It isn't. Fixed. What was the problem and the fix? Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 04:26, Scott Crosby schreef: What was the problem and the fix? The problem was that I assumed that after a succesful inflate nothing in my inflate buffer could have gone wrong. But I was actually placed on the end of that buffer. So basically protobuf pretends everything is 'ok' and just starts parsing. So the fix was just nicely to pass the start of the inflated data. I was coding quite defensive, but it seems protobuf itself still 'eats' memory. Given the presented 'max 64MB limits', not really an issue, but it is just duplicating a lot of memory, instead of using pointers to the input arrays. For whats worth, currently I'm able to decode the 'fileformat', doing deflate decompression. So still some stuff to go, but I understand the basics now... so I'm a happy farmer. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycE98ACgkQYH1+F2Rqwn2svwCghj5x+u77Dg6ZIO1Fb9IdP/4l jMUAn1Irj/uLFzdESAZMQ/MnJ8X/oQHO =9QVf -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 24-09-10 04:58, Stefan de Konink schreef: For whats worth, currently I'm able to decode the 'fileformat', doing deflate decompression. So still some stuff to go, but I understand the basics now... so I'm a happy farmer. Small status update, Dense Nodes implemented. Including XML rendering. Normal nodes, ways and relations remaining. Thingie will be finished later today. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkycMT0ACgkQYH1+F2Rqwn1krQCfVRAmO52YMg+RWvKZ1iuRldQB wPoAoI3t1O7h2LjdwpHj+C7DgCTwJGhD =38Kl -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
