This is an automated email from the git hooks/post-receive script. sebastic pushed a commit to branch upstream in repository osm2pgsql.
commit 0a8b63fb27b8084adec80eb729bbb98ad7628a21 Author: Bas Couwenberg <sebas...@xs4all.nl> Date: Sat Jul 11 10:20:57 2015 +0200 Imported Upstream version 0.88.0~rc1 --- .gitignore | 5 +- AUTHORS | 4 +- CONTRIBUTING.md | 102 ++++++++++++++ Makefile.am | 15 ++- README.md | 25 +--- configure.ac | 2 +- docs/migrations.md | 15 +++ geometry-builder.cpp | 57 ++++---- node-persistent-cache.cpp | 89 ++++++------ node-ram-cache.cpp | 53 +++++--- node-ram-cache.hpp | 21 ++- options.cpp | 14 +- pgsql-id-tracker.cpp | 153 --------------------- pgsql-id-tracker.hpp | 31 ----- tagtransform.cpp | 52 ++++--- tests/common-pg.cpp | 35 ++--- tests/common-pg.hpp | 2 + tests/regression-test.py | 4 +- tests/test-middle-flat.cpp | 112 ++++++++++++++++ tests/test-output-multi-line.cpp | 2 - tests/test-output-multi-point-multi-table.cpp | 2 - tests/test-output-multi-point.cpp | 2 - tests/test-output-multi-polygon.cpp | 2 - tests/test-output-pgsql-tablespace.cpp | 146 ++++++++++++++++++++ tests/test-output-pgsql-z_order.cpp | 186 ++++++++++++++++++++++++++ tests/test-output-pgsql.cpp | 103 +++++++++++++- tests/test_output_pgsql_z_order.osm | 75 +++++++++++ 27 files changed, 938 insertions(+), 371 deletions(-) diff --git a/.gitignore b/.gitignore index 6ca8a4d..a4cd9b6 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,7 @@ tests/.dirstamp tests/test-parse-xml2 tests/test-middle-ram tests/test-middle-pgsql +tests/test-middle-flat tests/test-pgsql-escape tests/test-parse-options tests/test-output-multi-tags @@ -55,10 +56,12 @@ tests/test-output-multi-point-multi-table tests/test-output-multi-polygon tests/test-output-multi-poly-trivial tests/test-output-pgsql +tests/test-output-pgsql-tablespace +tests/test-output-pgsql-z_order tests/test-expire-tiles tests/*.log tests/*.trs -tests/test_output_pgsql_area_way.flat.nodes.bin +tests/*.flat.nodes.bin .libs/ *.lo diff --git a/AUTHORS b/AUTHORS index af8498b..0f6bbfa 100644 --- a/AUTHORS +++ b/AUTHORS @@ -1,3 +1,3 @@ osm2pgsql was written by Jon Burgess, Artem Pavlenko, Martijn van Oosterhout -Sarah Hoffman, Kai Krueger, Frederik Ramm, Brian Quinion and other -OpenStreetMap project members. \ No newline at end of file +Sarah Hoffmann, Kai Krueger, Frederik Ramm, Brian Quinion, Matt Amos, +Kevin Kreiser, Paul Norman and other OpenStreetMap project members. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..27bb5ca --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,102 @@ +# Osm2pgsql contribution guidelines + +## Workflow + +We operate the "Fork & Pull" model explained at + +https://help.github.com/articles/using-pull-requests + +You should fork the project into your own repo, create a topic branch +there and then make one or more pull requests back to the openstreetmap repository. +Your pull requests will then be reviewed and discussed. + +## History + +To understand the osm2pgsql code, it helps to know some history on it. Osm2pgsql +was written in C in 2007 as a port of an older Python utility. In 2014 it was +ported to C++ by MapQuest and the last C version was released as 0.86.0. In it's +time, it has had varying contribution activity, including times with no +maintainer or active developers. + +Parts of the codebase still clearly show their C origin and could use rewriting +in modern C++, making use of data structures in the standard library. + +## Versioning + +Osm2pgsql uses a X.Y.Z version number, where Y tells you if you are on a stable +or development series. Like the Linux Kernel, even numbers are stable and +development versions are odd. + +Old branches and versions are not generally maintained. + +## Code style + +The current codebase is a mix of styles, but new code should be written in the +[K&R 1TBS style](https://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS) with +4 spaces indentation. Tabs should never be used in the C++ code. + +e.g. + +``` +int main(int argc, char *argv[]) +{ + ... + while (x == y) { + something(); + somethingelse(); + + if (some_error) { + do_correct(); + } else { + continue_as_usual(); + } + } + + finalthing(); + ... +} +``` + +Names should use underscores, not camel case, with class/struct names ending in `_t`. + +## Platforms targeted + +Ideally osm2pgsql should compile on Linux, OS X, FreeBSD and Windows. It is +actively tested on Debian, Ubuntu and FreeBSD by the maintainers. + +## Testing + +The code also comes with a suite of tests which can be run by +executing ``make check``. + +Most of these tests depend on being able to set up a database and run osm2pgsql +against it. You need to ensure that PostgreSQL is running and that your user is +a superuser of that system. To do that, run: + +```sh +sudo -u postgres createuser -s $USER +sudo mkdir -p /tmp/psql-tablespace +sudo chown postgres.postgres /tmp/psql-tablespace +psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres +``` + +Once this is all set up, all the tests should run (no SKIPs), and pass +(no FAILs). If you encounter a failure, you can find more information +by looking in the `test-suite.log`. If you find something which seems +to be a bug, please check to see if it is a known issue at +https://github.com/openstreetmap/osm2pgsql/issues and, if it's not +already known, report it there. + +If running the tests in a virtual machine, allocate sufficient disk space for a +20GB flat nodes file. + +### Performance Testing + +If performance testing with a full planet import is required, indicate what +needs testing in a pull request. + +## Maintainers + +The current maintainers of osm2pgsql are [Sarah Hoffmann](https://github.com/lonvia/) +and [Paul Norman](https://github.com/pnorman/). Sarah has more experience with +the gazetteer backend and Paul with the pgsql and multi backends. diff --git a/Makefile.am b/Makefile.am index 8ce3f4f..c91d408 100644 --- a/Makefile.am +++ b/Makefile.am @@ -32,7 +32,6 @@ libosm2pgsql_la_SOURCES = \ parse-pbf.cpp \ parse-xml2.cpp \ pgsql.cpp \ - pgsql-id-tracker.cpp \ processor-line.cpp \ processor-point.cpp \ processor-polygon.cpp \ @@ -51,6 +50,7 @@ check_PROGRAMS = \ tests/test-parse-xml2 \ tests/test-middle-ram \ tests/test-middle-pgsql \ + tests/test-middle-flat \ tests/test-output-multi-tags \ tests/test-output-multi-line \ tests/test-output-multi-line-storage \ @@ -59,6 +59,8 @@ check_PROGRAMS = \ tests/test-output-multi-polygon \ tests/test-output-multi-poly-trivial \ tests/test-output-pgsql \ + tests/test-output-pgsql-z_order \ + tests/test-output-pgsql-tablespace \ tests/test-pgsql-escape \ tests/test-parse-options \ tests/test-expire-tiles @@ -69,6 +71,8 @@ tests_test_middle_ram_SOURCES = tests/test-middle-ram.cpp tests/middle-tests.cpp tests_test_middle_ram_LDADD = libosm2pgsql.la tests_test_middle_pgsql_SOURCES = tests/test-middle-pgsql.cpp tests/middle-tests.cpp tests/common-pg.cpp tests_test_middle_pgsql_LDADD = libosm2pgsql.la +tests_test_middle_flat_SOURCES = tests/test-middle-flat.cpp tests/middle-tests.cpp tests/common-pg.cpp +tests_test_middle_flat_LDADD = libosm2pgsql.la tests_test_output_multi_tags_SOURCES = tests/test-output-multi-tags.cpp tests/common-pg.cpp tests_test_output_multi_tags_LDADD = libosm2pgsql.la tests_test_output_multi_line_SOURCES = tests/test-output-multi-line.cpp tests/common-pg.cpp @@ -85,6 +89,10 @@ tests_test_output_multi_poly_trivial_SOURCES = tests/test-output-multi-poly-triv tests_test_output_multi_poly_trivial_LDADD = libosm2pgsql.la tests_test_output_pgsql_SOURCES = tests/test-output-pgsql.cpp tests/common-pg.cpp tests_test_output_pgsql_LDADD = libosm2pgsql.la +tests_test_output_pgsql_tablespace_SOURCES = tests/test-output-pgsql-tablespace.cpp tests/common-pg.cpp +tests_test_output_pgsql_tablespace_LDADD = libosm2pgsql.la +tests_test_output_pgsql_z_order_SOURCES = tests/test-output-pgsql-z_order.cpp tests/common-pg.cpp +tests_test_output_pgsql_z_order_LDADD = libosm2pgsql.la tests_test_pgsql_escape_SOURCES = tests/test-pgsql-escape.cpp tests_test_pgsql_escape_LDADD = libosm2pgsql.la tests_test_parse_options_SOURCES = tests/test-parse-options.cpp @@ -92,6 +100,8 @@ tests_test_parse_options_LDADD = libosm2pgsql.la tests_test_expire_tiles_SOURCES = tests/test-expire-tiles.cpp tests_test_expire_tiles_LDADD = libosm2pgsql.la +MOSTLYCLEANFILES = tests/test_middle_flat.flat.nodes.bin tests/test_output_pgsql_area_way.flat.nodes.bin + TESTS = $(check_PROGRAMS) tests/regression-test.sh TEST_EXTENSIONS = .sh SH_LOG_COMPILER = sh @@ -138,6 +148,7 @@ osm2pgsql_LDADD += $(GLOBAL_LDFLAGS) tests_test_parse_xml2_LDADD += $(GLOBAL_LDFLAGS) tests_test_middle_ram_LDADD += $(GLOBAL_LDFLAGS) tests_test_middle_pgsql_LDADD += $(GLOBAL_LDFLAGS) +tests_test_middle_flat_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_multi_tags_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_multi_line_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_multi_line_storage_LDADD += $(GLOBAL_LDFLAGS) @@ -146,6 +157,8 @@ tests_test_output_multi_point_multi_table_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_multi_polygon_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_multi_poly_trivial_LDADD += $(GLOBAL_LDFLAGS) tests_test_output_pgsql_LDADD += $(GLOBAL_LDFLAGS) +tests_test_output_pgsql_tablespace_LDADD += $(GLOBAL_LDFLAGS) +tests_test_output_pgsql_z_order_LDADD += $(GLOBAL_LDFLAGS) tests_test_pgsql_escape_LDADD += $(GLOBAL_LDFLAGS) tests_test_parse_options_LDADD += $(GLOBAL_LDFLAGS) tests_test_expire_tiles_LDADD += $(GLOBAL_LDFLAGS) diff --git a/README.md b/README.md index f6bdd68..b601b8c 100644 --- a/README.md +++ b/README.md @@ -143,33 +143,12 @@ postgres tables instead of those provided in the pgsql backend. Any questions should be directed at the osm dev list http://wiki.openstreetmap.org/index.php/Mailing_lists -## Testing ## - -The code also comes with a suite of tests which can be run by -executing ``make check``. - -Some of these tests depend on being able to set up a database and run -osm2pgsql against it. You need to ensure that PostgreSQL is running -and that your user is a superuser of that system. To do that, run: - -```sh -sudo -u postgres createuser -s $USER -sudo mkdir -p /tmp/psql-tablespace -sudo chown postgres.postgres /tmp/psql-tablespace -psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres -``` - -Once this is all set up, all the tests should run (no SKIPs), and pass -(no FAILs). If you encounter a failure, you can find more information -by looking in the `test-suite.log`. If you find something which seems -to be a bug, please check to see if it is a known issue at -https://github.com/openstreetmap/osm2pgsql/issues and, if it's not -already known, report it there. - ## Contributing ## We welcome contributions to osm2pgsql. If you would like to report an issue, please use the [issue tracker on GitHub](https://github.com/openstreetmap/osm2pgsql/issues). +More information can be found in [CONTRIBUTING.md](CONTRIBUTING.md). + General queries can be sent to the tile-serving@ or dev@ [mailing lists](http://wiki.openstreetmap.org/wiki/Mailing_lists). diff --git a/configure.ac b/configure.ac index 278de33..a545f9d 100644 --- a/configure.ac +++ b/configure.ac @@ -1,5 +1,5 @@ dnl Process this file with autoconf to produce a configure script. -AC_INIT(osm2pgsql, 0.87.4) +AC_INIT(osm2pgsql, 0.88.0-RC1) dnl Required autoconf version AC_PREREQ(2.61) diff --git a/docs/migrations.md b/docs/migrations.md index 10457c8..af1c70a 100644 --- a/docs/migrations.md +++ b/docs/migrations.md @@ -4,6 +4,21 @@ Some osm2pgsql changes have slightly changed the database schema it expects. If updating an old database, a migration may be needed. The migrations here assume the default `planet_osm` prefix. +## 0.88.0 z_order changes ## + +0.88.0 z_order logic was changed, requuiring an increase in z_order values. To +migrate to the new range of values, run + +```sql +UPDATE planet_osm_line SET z_order = z_order * 10; +UPDATE planet_osm_roads SET z_order = z_order * 10; +``` + +This will not apply the new logic, but will get the existing z_orders in the right +group of 100 for the new logic. + +If not using osm2pgsql z_orders, this change may be ignored. + ## 0.87.0 pending removal ## 0.87.0 moved the in-database tracking of pending ways and relations to diff --git a/geometry-builder.cpp b/geometry-builder.cpp index 143a1cc..5c4a6b5 100644 --- a/geometry-builder.cpp +++ b/geometry-builder.cpp @@ -81,9 +81,9 @@ struct polygondata }; struct polygondata_comparearea { - bool operator()(const polygondata& lhs, const polygondata& rhs) { - return lhs.area > rhs.area; - } + bool operator()(const polygondata& lhs, const polygondata& rhs) { + return lhs.area > rhs.area; + } }; } // anonymous namespace @@ -190,25 +190,25 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t // length of the line in `segment` over the `split_at` distance. if (distance + delta > split_at) { - const size_t splits = std::floor((distance + delta) / split_at); - // use the splitting distance to split the current segment up - // into as many parts as necessary to keep each part below - // the `split_at` distance. - for (size_t i = 0; i < splits; ++i) { - double frac = (double(i + 1) * split_at - distance) / delta; - const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x, - frac * (this_pt.y - prev_pt.y) + prev_pt.y); - segment->add(interpolated); - geom_ptr geom = geom_ptr(gf.createLineString(segment.release())); + const size_t splits = std::floor((distance + delta) / split_at); + // use the splitting distance to split the current segment up + // into as many parts as necessary to keep each part below + // the `split_at` distance. + for (size_t i = 0; i < splits; ++i) { + double frac = (double(i + 1) * split_at - distance) / delta; + const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x, + frac * (this_pt.y - prev_pt.y) + prev_pt.y); + segment->add(interpolated); + geom_ptr geom = geom_ptr(gf.createLineString(segment.release())); - //copy of an empty one should be cheapest - wkts->push_back(geometry_builder::wkt_t()); - //then we set on the one we already have - wkts->back().geom = writer.write(geom.get()); - wkts->back().area = 0; + //copy of an empty one should be cheapest + wkts->push_back(geometry_builder::wkt_t()); + //then we set on the one we already have + wkts->back().geom = writer.write(geom.get()); + wkts->back().area = 0; - segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2)); - segment->add(interpolated); + segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2)); + segment->add(interpolated); } // reset the distance based on the final splitting point for // the next iteration. @@ -255,12 +255,12 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t } int geometry_builder::parse_wkt(const char * wkt, multinodelist_t &nodes, int *polygon) { - GeometryFactory gf; - WKTReader reader(&gf); - std::string wkt_string(wkt); - GeometryCollection * gc; - CoordinateSequence * coords; - size_t num_geometries; + GeometryFactory gf; + WKTReader reader(&gf); + std::string wkt_string(wkt); + GeometryCollection * gc; + CoordinateSequence * coords; + size_t num_geometries; *polygon = 0; try { @@ -320,7 +320,6 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel maybe_wkts_t wkts(new std::vector<geometry_builder::wkt_t>); - try { for (multinodelist_t::const_iterator it = xnodes.begin(); it != xnodes.end(); ++it) { @@ -422,7 +421,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel } } } - pgf.destroy(preparedtoplevelpolygon); + pgf.destroy(preparedtoplevelpolygon); } // polys now is a list of polygons tagged with which ones are inside each other @@ -608,7 +607,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_both(const multinodelist_ } else { - //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl; + //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl; double distance = 0; std::auto_ptr<CoordinateSequence> segment; segment = std::auto_ptr<CoordinateSequence>(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2)); diff --git a/node-persistent-cache.cpp b/node-persistent-cache.cpp index 61a4804..3689ce7 100644 --- a/node-persistent-cache.cpp +++ b/node-persistent-cache.cpp @@ -49,10 +49,10 @@ void node_persistent_cache::writeout_dirty_nodes() { for (int i = 0; i < READ_NODE_CACHE_SIZE; i++) { - if (readNodeBlockCache[i].dirty) + if (readNodeBlockCache[i].dirty()) { if (lseek64(node_cache_fd, - (readNodeBlockCache[i].block_offset + ((osmid_t) readNodeBlockCache[i].block_offset << READ_NODE_BLOCK_SHIFT) * sizeof(ramNode) + sizeof(persistentCacheHeader), @@ -70,7 +70,7 @@ void node_persistent_cache::writeout_dirty_nodes() util::exit_nicely(); } } - readNodeBlockCache[i].dirty = 0; + readNodeBlockCache[i].reset_used(); } } @@ -85,9 +85,9 @@ int node_persistent_cache::replace_block() for (int i = 0; i < READ_NODE_CACHE_SIZE; i++) { - if (readNodeBlockCache[i].used < min_used) + if (readNodeBlockCache[i].used() < min_used) { - min_used = readNodeBlockCache[i].used; + min_used = readNodeBlockCache[i].used(); block_id = i; } } @@ -95,9 +95,9 @@ int node_persistent_cache::replace_block() { for (int i = 0; i < READ_NODE_CACHE_SIZE; i++) { - if (readNodeBlockCache[i].used > 1) + if (readNodeBlockCache[i].used() > 1) { - readNodeBlockCache[i].used--; + readNodeBlockCache[i].dec_used(); } } } @@ -214,10 +214,10 @@ int node_persistent_cache::load_block(osmid_t block_offset) { const int block_id = replace_block(); - if (readNodeBlockCache[block_id].dirty) + if (readNodeBlockCache[block_id].dirty()) { if (lseek64(node_cache_fd, - (readNodeBlockCache[block_id].block_offset + ((osmid_t) readNodeBlockCache[block_id].block_offset << READ_NODE_BLOCK_SHIFT) * sizeof(ramNode) + sizeof(struct persistentCacheHeader), SEEK_SET) < 0) { fprintf(stderr, "Failed to seek to correct position in node cache: %s\n", @@ -232,13 +232,21 @@ int node_persistent_cache::load_block(osmid_t block_offset) strerror(errno)); util::exit_nicely(); } - readNodeBlockCache[block_id].dirty = 0; + readNodeBlockCache[block_id].reset_used(); } - remove_from_cache_idx(readNodeBlockCache[block_id].block_offset); - new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE]; + if (readNodeBlockCache[block_id].nodes) { + remove_from_cache_idx((osmid_t) readNodeBlockCache[block_id].block_offset); + new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE]; + } else { + readNodeBlockCache[block_id].nodes = new ramNode[READ_NODE_BLOCK_SIZE]; + if (!readNodeBlockCache[block_id].nodes) { + fprintf(stderr, "Out of memory: Failed to allocate node read cache\n"); + util::exit_nicely(); + } + } readNodeBlockCache[block_id].block_offset = block_offset; - readNodeBlockCache[block_id].used = READ_NODE_CACHE_SIZE; + readNodeBlockCache[block_id].set_used(READ_NODE_CACHE_SIZE); /* Make sure the node cache is correctly initialised for the block that will be read */ if (cacheHeader.max_initialised_id @@ -289,7 +297,7 @@ void node_persistent_cache::nodes_set_create_writeout_block() * node cache file in buffer cache therefore duplicates the data wasting 16GB of ram. * Therefore tell the OS not to cache the node-persistent-cache during initial import. * */ - if (sync_file_range(node_cache_fd, writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + + if (sync_file_range(node_cache_fd, (osmid_t) writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode), SYNC_FILE_RANGE_WRITE) < 0) { fprintf(stderr, "Info: Sync_file_range writeout has an issue. This shouldn't be anything to worry about.: %s\n", @@ -297,7 +305,7 @@ void node_persistent_cache::nodes_set_create_writeout_block() }; if (writeNodeBlock.block_offset > 16) { - if(sync_file_range(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + + if(sync_file_range(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode), SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER) < 0) { fprintf(stderr, "Info: Sync_file_range block has an issue. This shouldn't be anything to worry about.: %s\n", @@ -305,7 +313,7 @@ void node_persistent_cache::nodes_set_create_writeout_block() } #ifdef HAVE_POSIX_FADVISE - if (posix_fadvise(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + + if (posix_fadvise(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) + sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode), POSIX_FADV_DONTNEED) !=0 ) { fprintf(stderr, "Info: Posix_fadvise failed. This shouldn't be anything to worry about.: %s\n", strerror(errno)); @@ -320,23 +328,22 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon) assert(!append_mode); assert(!read_mode); - osmid_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT; + int32_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT; if (writeNodeBlock.block_offset != block_offset) { - if (writeNodeBlock.dirty) + if (writeNodeBlock.dirty()) { nodes_set_create_writeout_block(); - writeNodeBlock.dirty = 0; /* After writing out the node block, the file pointer is at the next block level */ writeNodeBlock.block_offset++; - cacheHeader.max_initialised_id = (writeNodeBlock.block_offset + cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset << WRITE_NODE_BLOCK_SHIFT) - 1; } if (writeNodeBlock.block_offset > block_offset) { fprintf(stderr, - "ERROR: Block_offset not in sequential order: %" PRIdOSMID "%" PRIdOSMID "\n", + "ERROR: Block_offset not in sequential order: %d %d\n", writeNodeBlock.block_offset, block_offset); util::exit_nicely(); } @@ -353,7 +360,7 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon) } writeNodeBlock.nodes[id & WRITE_NODE_BLOCK_MASK] = ramNode(lon, lat); - writeNodeBlock.dirty = 1; + writeNodeBlock.set_dirty(); return 0; } @@ -373,8 +380,8 @@ int node_persistent_cache::set_append(osmid_t id, double lat, double lon) readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode(); else readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode(lon, lat); - readNodeBlockCache[block_id].used++; - readNodeBlockCache[block_id].dirty = 1; + readNodeBlockCache[block_id].inc_used(); + readNodeBlockCache[block_id].set_dirty(); return 1; } @@ -399,7 +406,7 @@ int node_persistent_cache::get(osmNode *out, osmid_t id) block_id = load_block(block_offset); } - readNodeBlockCache[block_id].used++; + readNodeBlockCache[block_id].inc_used(); if (!readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK].is_valid()) return 1; @@ -451,13 +458,12 @@ void node_persistent_cache::set_read_mode() if (read_mode) return; - if (writeNodeBlock.dirty > 0) { + if (writeNodeBlock.dirty()) { assert(!append_mode); - fprintf(stderr, "Switching to read mode\n"); nodes_set_create_writeout_block(); - writeNodeBlock.dirty = 0; + writeNodeBlock.reset_used(); writeNodeBlock.block_offset++; - cacheHeader.max_initialised_id = (writeNodeBlock.block_offset + cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset << WRITE_NODE_BLOCK_SHIFT) - 1; /* write out the header */ @@ -475,8 +481,6 @@ void node_persistent_cache::set_read_mode() } read_mode = true; - - fprintf(stderr, "Switching to read mode done\n"); } node_persistent_cache::node_persistent_cache(const options_t *options, int append, @@ -484,7 +488,6 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen : node_cache_fd(0), node_cache_fname(NULL), append_mode(0), cacheHeader(), writeNodeBlock(), readNodeBlockCache(NULL), read_mode(ro), ram_cache(ptr) { - int i, err; append_mode = append; if (options->flat_node_file) { node_cache_fname = options->flat_node_file->c_str(); @@ -533,12 +536,10 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen }; writeNodeBlock.block_offset = 0; - writeNodeBlock.dirty = 0; - writeNodeBlock.nodes = 0; if (!read_mode) { - + int err; #ifdef HAVE_POSIX_FALLOCATE if ((err = posix_fallocate(node_cache_fd, 0, sizeof(ramNode) * MAXIMUM_INITIAL_ID)) != 0) @@ -606,27 +607,16 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen fprintf(stderr,"Maximum node in persistent node cache: %" PRIdOSMID "\n", cacheHeader.max_initialised_id); - readNodeBlockCache = new ramNodeBlock [READ_NODE_CACHE_SIZE]; + readNodeBlockCache = new ramNodeBlock[READ_NODE_CACHE_SIZE]; if (!readNodeBlockCache) { fprintf(stderr, "Out of memory: Failed to allocate node read cache\n"); util::exit_nicely(); } - for (i = 0; i < READ_NODE_CACHE_SIZE; i++) - { - readNodeBlockCache[i].nodes = new ramNode[READ_NODE_BLOCK_SIZE]; - if (!readNodeBlockCache[i].nodes) { - fprintf(stderr, "Out of memory: Failed to allocate node read cache\n"); - util::exit_nicely(); - } - readNodeBlockCache[i].block_offset = -1; - readNodeBlockCache[i].used = 0; - readNodeBlockCache[i].dirty = 0; - } } node_persistent_cache::~node_persistent_cache() { - if (writeNodeBlock.dirty > 0) + if (writeNodeBlock.dirty()) nodes_set_create_writeout_block(); writeout_dirty_nodes(); @@ -659,7 +649,8 @@ node_persistent_cache::~node_persistent_cache() if (readNodeBlockCache) { for (int i = 0; i < READ_NODE_CACHE_SIZE; i++) { - delete[] readNodeBlockCache[i].nodes; + if (readNodeBlockCache[i].nodes) + delete[] readNodeBlockCache[i].nodes; } delete[] readNodeBlockCache; } diff --git a/node-ram-cache.cpp b/node-ram-cache.cpp index 167f6b7..23f8d24 100644 --- a/node-ram-cache.cpp +++ b/node-ram-cache.cpp @@ -60,7 +60,7 @@ int ramNode::scale; -static int id2block(osmid_t id) +static int32_t id2block(osmid_t id) { /* + NUM_BLOCKS/2 allows for negative IDs */ return (id >> BLOCK_SHIFT) + NUM_BLOCKS/2; @@ -71,7 +71,7 @@ static int id2offset(osmid_t id) return id & (PER_BLOCK-1); } -static osmid_t block2id(int block, int offset) +static osmid_t block2id(int32_t block, int offset) { return (((osmid_t) block - NUM_BLOCKS/2) << BLOCK_SHIFT) + (osmid_t) offset; } @@ -84,7 +84,7 @@ void node_ram_cache::percolate_up( int pos ) while( i > 0 ) { int parent = (i-1)>>1; - if( queue[i]->used < queue[parent]->used ) + if( queue[i]->used() < queue[parent]->used() ) { Swap( queue[i], queue[parent] ) i = parent; @@ -108,7 +108,11 @@ ramNode *node_ram_cache::next_chunk() { int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) { - if ((sizeSparseTuples > maxSparseTuples) || ( cacheUsed > cacheSize)) { + // Sparse cache depends on ordered nodes, reject out-of-order ids. + // Also check that there is still space. + if ((maxSparseId && id < maxSparseId) + || (sizeSparseTuples > maxSparseTuples) + || ( cacheUsed > cacheSize)) { if ((allocStrategy & ALLOC_LOSSY) > 0) return 1; else { @@ -116,6 +120,7 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) { util::exit_nicely(); } } + maxSparseId = id; sparseBlock[sizeSparseTuples].id = id; sparseBlock[sizeSparseTuples].coord = coord; @@ -126,9 +131,8 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) { } int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { - int block = id2block(id); - int offset = id2offset(id); - int i = 0; + int32_t const block = id2block(id); + int const offset = id2offset(id); if (maxBlocks == 0) return 1; @@ -143,7 +147,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { * to store it in dense representation. If not, push all elements of the block * to the sparse node cache and reuse memory of the previous block for the current block */ if ( ((allocStrategy & ALLOC_SPARSE) == 0) || - ((queue[usedBlocks - 1]->used / (double)(1<< BLOCK_SHIFT)) > + ((queue[usedBlocks - 1]->used() / (double)(1<< BLOCK_SHIFT)) > (sizeof(ramNode) / (double)sizeof(ramNodeID)))) { /* Block has reached the level to keep it in dense representation */ /* We've just finished with the previous block, so we need to percolate it up the queue to its correct position */ @@ -152,7 +156,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { blocks[block].nodes = next_chunk(); } else { /* previous block was not dense enough, so push it into the sparse node cache instead */ - for (i = 0; i < (1 << BLOCK_SHIFT); i++) { + for (int i = 0; i < (1 << BLOCK_SHIFT); i++) { if (queue[usedBlocks -1]->nodes[i].is_valid()) { set_sparse(block2id(queue[usedBlocks - 1]->block_offset, i), queue[usedBlocks -1]->nodes[i]); @@ -160,7 +164,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { } } /* reuse previous block, as its content is now in the sparse representation */ - storedNodes -= queue[usedBlocks - 1]->used; + storedNodes -= queue[usedBlocks - 1]->used(); blocks[block].nodes = queue[usedBlocks - 1]->nodes; blocks[queue[usedBlocks - 1]->block_offset].nodes = NULL; usedBlocks--; @@ -170,7 +174,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { blocks[block].nodes = next_chunk(); } - blocks[block].used = 0; + blocks[block].reset_used(); blocks[block].block_offset = block; if (!blocks[block].nodes) { fprintf(stderr, "Error allocating nodes\n"); @@ -194,17 +198,17 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { * current head of the tree down to the right level to restore the * priority queue invariant. Upto log(maxBlocks) iterations */ - i=0; + int i = 0; while( 2*i+1 < usedBlocks - 1 ) { - if( queue[2*i+1]->used <= queue[2*i+2]->used ) { - if( queue[i]->used > queue[2*i+1]->used ) { + if( queue[2*i+1]->used() <= queue[2*i+2]->used() ) { + if( queue[i]->used() > queue[2*i+1]->used() ) { Swap( queue[i], queue[2*i+1] ); i = 2*i+1; } else break; } else { - if( queue[i]->used > queue[2*i+2]->used ) { + if( queue[i]->used() > queue[2*i+2]->used() ) { Swap( queue[i], queue[2*i+2] ); i = 2*i+2; } else @@ -213,13 +217,13 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { } /* Now the head of the queue is the smallest, so it becomes our replacement candidate */ blocks[block].nodes = queue[0]->nodes; - blocks[block].used = 0; + blocks[block].reset_used(); new(blocks[block].nodes) ramNode[PER_BLOCK]; /* Clear old head block and point to new block */ - storedNodes -= queue[0]->used; + storedNodes -= queue[0]->used(); queue[0]->nodes = NULL; - queue[0]->used = 0; + queue[0]->reset_used(); queue[0] = &blocks[block]; } } else { @@ -243,7 +247,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) { } blocks[block].nodes[offset] = coord; - blocks[block].used++; + blocks[block].inc_used(); storedNodes++; return 0; } @@ -275,8 +279,8 @@ int node_ram_cache::get_sparse(osmNode *out, osmid_t id) { } int node_ram_cache::get_dense(osmNode *out, osmid_t id) { - int block = id2block(id); - int offset = id2offset(id); + int32_t const block = id2block(id); + int const offset = id2offset(id); if (!blocks[block].nodes) return 1; @@ -294,7 +298,7 @@ int node_ram_cache::get_dense(osmNode *out, osmid_t id) { node_ram_cache::node_ram_cache( int strategy, int cacheSizeMB, int fixpointscale ) : allocStrategy(ALLOC_DENSE), blocks(NULL), usedBlocks(0), maxBlocks(0), blockCache(NULL), queue(NULL), sparseBlock(NULL), - maxSparseTuples(0), sizeSparseTuples(0), cacheUsed(0), + maxSparseTuples(0), sizeSparseTuples(0), maxSparseId(0), cacheUsed(0), cacheSize(0), storedNodes(0), totalNodes(0), nodesCacheHits(0), nodesCacheLookups(0), warn_node_order(0) { @@ -390,6 +394,11 @@ node_ram_cache::~node_ram_cache() { } int node_ram_cache::set(osmid_t id, double lat, double lon, const taglist_t &) { + if ((id > 0 && id >> BLOCK_SHIFT >> 32) || (id < 0 && ~id >> BLOCK_SHIFT >> 32 )) { + fprintf(stderr, "\nAbsolute node IDs must not be larger than %lld (got %lld)\n", + 1ULL << 42, (long long) id); + util::exit_nicely(); + } totalNodes++; /* if ALLOC_DENSE and ALLOC_SPARSE are set, send it through * ram_nodes_set_dense. If a block is non dense, it will automatically diff --git a/node-ram-cache.hpp b/node-ram-cache.hpp index 0404904..c292c2a 100644 --- a/node-ram-cache.hpp +++ b/node-ram-cache.hpp @@ -83,11 +83,23 @@ struct ramNodeID { ramNode coord; }; -struct ramNodeBlock { +class ramNodeBlock { +public: + ramNodeBlock() : nodes(NULL), block_offset(-1), _used(0) {} + + void set_dirty() { _used |= 1; } + bool dirty() const { return _used & 1; } + + void reset_used() { _used = 0; } + void inc_used() { _used += 2; } + void dec_used() { _used -= 2; } + void set_used(int used) { _used = (used << 1) || (_used & 1); } + int used() const { return _used >> 1; } + ramNode *nodes; - osmid_t block_offset; - int used; - int dirty; + int32_t block_offset; +private: + int32_t _used; // 0-bit indicates dirty }; struct node_ram_cache : public boost::noncopyable @@ -120,6 +132,7 @@ private: ramNodeID *sparseBlock; int64_t maxSparseTuples; int64_t sizeSparseTuples; + osmid_t maxSparseId; int64_t cacheUsed, cacheSize; osmid_t storedNodes, totalNodes; diff --git a/options.cpp b/options.cpp index 19462b4..8c2ed03 100644 --- a/options.cpp +++ b/options.cpp @@ -385,7 +385,7 @@ options_t options_t::parse(int argc, char *argv[]) break; case 'k': if (options.hstore_mode != HSTORE_NONE) { - throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n"); + throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n"); } options.hstore_mode = HSTORE_NORM; break; @@ -394,7 +394,7 @@ options_t options_t::parse(int argc, char *argv[]) break; case 'j': if (options.hstore_mode != HSTORE_NONE) { - throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n"); + throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n"); } options.hstore_mode = HSTORE_ALL; break; @@ -425,7 +425,7 @@ options_t options_t::parse(int argc, char *argv[]) else if (strcmp(optarg, "optimized") == 0) options.alloc_chunkwise = ALLOC_DENSE | ALLOC_SPARSE; else { - throw std::runtime_error((boost::format("ERROR: Unrecognized cache strategy %1%.\n") % optarg).str()); + throw std::runtime_error((boost::format("Unrecognized cache strategy %1%.\n") % optarg).str()); } break; case 205: @@ -481,11 +481,15 @@ options_t options_t::parse(int argc, char *argv[]) } if (options.append && options.create) { - throw std::runtime_error("Error: --append and --create options can not be used at the same time!\n"); + throw std::runtime_error("--append and --create options can not be used at the same time!\n"); + } + + if (options.append && !options.slim) { + throw std::runtime_error("--append can only be used with slim mode!\n"); } if (options.droptemp && !options.slim) { - throw std::runtime_error("Error: --drop only makes sense with --slim.\n"); + throw std::runtime_error("--drop only makes sense with --slim.\n"); } if (options.unlogged && !options.create) { diff --git a/pgsql-id-tracker.cpp b/pgsql-id-tracker.cpp deleted file mode 100644 index bcd54af..0000000 --- a/pgsql-id-tracker.cpp +++ /dev/null @@ -1,153 +0,0 @@ -#include "pgsql-id-tracker.hpp" - -#include <cassert> -#include <cstdio> -#include <limits> -#include <libpq-fe.h> -#include <boost/format.hpp> - -#include "osmtypes.hpp" -#include "pgsql.hpp" -#include "util.hpp" - -struct pgsql_id_tracker::pimpl { - pimpl(const std::string &conninfo, - const std::string &prefix, - const std::string &type, - bool owns_table); - ~pimpl(); - - PGconn *conn; - std::string table_name; - bool owns_table; - osmid_t old_id; -}; - -pgsql_id_tracker::pimpl::pimpl(const std::string &conninfo, - const std::string &prefix, - const std::string &type, - bool owns_table_) - : conn(PQconnectdb(conninfo.c_str())), - table_name((boost::format("%1%_%2%") % prefix % type).str()), - owns_table(owns_table_), - old_id(0) { - if (PQstatus(conn) != CONNECTION_OK) { - fprintf(stderr, "Connection to database failed: %s\n", PQerrorMessage(conn)); - util::exit_nicely(); - } - if (owns_table) { - pgsql_exec(conn, PGRES_COMMAND_OK, - "DROP TABLE IF EXISTS \"%s\"", - table_name.c_str()); - pgsql_exec(conn, PGRES_COMMAND_OK, - "CREATE TABLE \"%s\" (id " POSTGRES_OSMID_TYPE ")", - table_name.c_str()); - } - pgsql_exec(conn, PGRES_COMMAND_OK, - "PREPARE set_mark(" POSTGRES_OSMID_TYPE ") AS INSERT INTO \"%s\" (id) " - "SELECT $1 WHERE NOT EXISTS (SELECT id FROM \"%s\" WHERE id = $1)", - table_name.c_str(), table_name.c_str()); - pgsql_exec(conn, PGRES_COMMAND_OK, - "PREPARE get_mark(" POSTGRES_OSMID_TYPE ") AS SELECT id FROM \"%s\" " - "WHERE id = $1", - table_name.c_str()); - pgsql_exec(conn, PGRES_COMMAND_OK, - "PREPARE get_min AS SELECT min(id) AS id FROM \"%s\"", - table_name.c_str()); - pgsql_exec(conn, PGRES_COMMAND_OK, - "PREPARE drop_mark(" POSTGRES_OSMID_TYPE ") AS DELETE FROM \"%s\" " - "WHERE id = $1", - table_name.c_str()); - pgsql_exec(conn, PGRES_COMMAND_OK, "BEGIN"); -} - -pgsql_id_tracker::pimpl::~pimpl() { - if (conn) { - pgsql_exec(conn, PGRES_COMMAND_OK, "COMMIT"); - if (owns_table) { - pgsql_exec(conn, PGRES_COMMAND_OK, "DROP TABLE \"%s\"", table_name.c_str()); - } - PQfinish(conn); - } - conn = NULL; -} - -pgsql_id_tracker::pgsql_id_tracker(const std::string &conninfo, - const std::string &prefix, - const std::string &type, - bool owns_table) - : impl() { - impl.reset(new pimpl(conninfo, prefix, type, owns_table)); -} - -pgsql_id_tracker::~pgsql_id_tracker() { -} - -void pgsql_id_tracker::mark(osmid_t id) { - char tmp[16]; - char const *paramValues[1]; - - snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id); - paramValues[0] = tmp; - - pgsql_execPrepared(impl->conn, "set_mark", 1, paramValues, PGRES_COMMAND_OK); -} - -bool pgsql_id_tracker::is_marked(osmid_t id) { - char tmp[16]; - char const *paramValues[1] = {NULL}; - PGresult *result = NULL; - - snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id); - paramValues[0] = tmp; - - result = pgsql_execPrepared(impl->conn, "get_mark", 1, paramValues, PGRES_TUPLES_OK); - bool done = PQntuples(result) > 0; - PQclear(result); - return done; -} - -osmid_t pgsql_id_tracker::pop_mark() { - osmid_t id = std::numeric_limits<osmid_t>::max(); - PGresult *result = NULL; - - result = pgsql_execPrepared(impl->conn, "get_min", 0, NULL, PGRES_TUPLES_OK); - if ((PQntuples(result) == 1) && - (PQgetisnull(result, 0, 0) == 0)) { - id = strtoosmid(PQgetvalue(result, 0, 0), NULL, 10); - } - - PQclear(result); - - if (id != std::numeric_limits<osmid_t>::max()) { - unmark(id); - } - - assert((id > impl->old_id) || (id == std::numeric_limits<osmid_t>::max())); - impl->old_id = id; - - return id; -} - -void pgsql_id_tracker::unmark(osmid_t id) { - char tmp[16]; - char const *paramValues[1] = {NULL}; - - snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id); - paramValues[0] = tmp; - - pgsql_execPrepared(impl->conn, "drop_mark", 1, paramValues, PGRES_COMMAND_OK); -} - -void pgsql_id_tracker::commit() { - if (impl->owns_table) { - pgsql_exec(impl->conn, PGRES_COMMAND_OK, "CREATE INDEX ON \"%s\" (id)", impl->table_name.c_str()); - } - pgsql_exec(impl->conn, PGRES_COMMAND_OK, "COMMIT"); - pgsql_exec(impl->conn, PGRES_COMMAND_OK, "BEGIN"); -} - -void pgsql_id_tracker::force_release() { - impl->owns_table = false; - impl->conn = NULL; -} diff --git a/pgsql-id-tracker.hpp b/pgsql-id-tracker.hpp deleted file mode 100644 index 4a140ee..0000000 --- a/pgsql-id-tracker.hpp +++ /dev/null @@ -1,31 +0,0 @@ -#ifndef PGSQL_ID_TRACKER_HPP -#define PGSQL_ID_TRACKER_HPP - -#include "id-tracker.hpp" - -#include <string> -#include <boost/smart_ptr/scoped_ptr.hpp> - -struct pgsql_id_tracker : public id_tracker { - pgsql_id_tracker(const std::string &conninfo, - const std::string &prefix, - const std::string &type, - bool owns_table); - ~pgsql_id_tracker(); - - void mark(osmid_t id); - bool is_marked(osmid_t id); - - osmid_t pop_mark(); - - void commit(); - void force_release(); // to avoid brain-damages with fork() - -private: - void unmark(osmid_t id); - - struct pimpl; - boost::scoped_ptr<pimpl> impl; -}; - -#endif /* PGSQL_ID_TRACKER_HPP */ diff --git a/tagtransform.cpp b/tagtransform.cpp index 208b33c..dd7d9fc 100644 --- a/tagtransform.cpp +++ b/tagtransform.cpp @@ -25,20 +25,33 @@ static const struct { const char *highway; int roads; } layers[] = { - { 3, "minor", 0 }, - { 3, "road", 0 }, - { 3, "unclassified", 0 }, - { 3, "residential", 0 }, - { 4, "tertiary_link", 0 }, - { 4, "tertiary", 0 }, - { 6, "secondary_link",1 }, - { 6, "secondary", 1 }, - { 7, "primary_link", 1 }, - { 7, "primary", 1 }, - { 8, "trunk_link", 1 }, - { 8, "trunk", 1 }, - { 9, "motorway_link", 1 }, - { 9, "motorway", 1 } + { 1, "proposed", 0 }, + { 2, "construction", 0 }, + { 10, "steps", 0 }, + { 10, "cycleway", 0 }, + { 10, "bridleway", 0 }, + { 10, "footway", 0 }, + { 10, "path", 0 }, + { 11, "track", 0 }, + { 15, "service", 0 }, + + { 24, "tertiary_link", 0 }, + { 25, "secondary_link",1 }, + { 27, "primary_link", 1 }, + { 28, "trunk_link", 1 }, + { 29, "motorway_link", 1 }, + + { 30, "raceway", 0 }, + { 31, "pedestrian", 0 }, + { 32, "living_street", 0 }, + { 33, "road", 0 }, + { 33, "unclassified", 0 }, + { 33, "residential", 0 }, + { 34, "tertiary", 0 }, + { 36, "secondary", 1 }, + { 37, "primary", 1 }, + { 38, "trunk", 1 }, + { 39, "motorway", 1 } }; static const unsigned int nLayers = (sizeof(layers)/sizeof(*layers)); @@ -56,12 +69,11 @@ void add_z_order(taglist_t &tags, int *roads) int z_order = 0; int l = layer ? strtol(layer->c_str(), NULL, 10) : 0; - z_order = 10 * l; + z_order = 100 * l; *roads = 0; if (highway) { for (unsigned i = 0; i < nLayers; i++) { - //if (layers[i].highway == *highway) { if (!strcmp(layers[i].highway, highway->c_str())) { z_order += layers[i].offset; *roads = layers[i].roads; @@ -71,7 +83,7 @@ void add_z_order(taglist_t &tags, int *roads) } if (railway && !railway->empty()) { - z_order += 5; + z_order += 35; *roads = 1; } /* Administrative boundaries are rendered at low zooms so we prefer to use the roads table */ @@ -79,10 +91,10 @@ void add_z_order(taglist_t &tags, int *roads) *roads = 1; if (bridge) - z_order += 10; + z_order += 100; if (tunnel) - z_order -= 10; + z_order -= 100; char z[13]; snprintf(z, sizeof(z), "%d", z_order); @@ -118,7 +130,7 @@ unsigned int c_filter_rel_member_tags(const taglist_t &rel_tags, if (is_route && (it->key == "name")) out_tags.push_dedupe(tag("route_name", it->value)); //copy all other tags except for "type" - else if (it->key != "type") + if (it->key != "type") out_tags.push_dedupe(*it); } diff --git a/tests/common-pg.cpp b/tests/common-pg.cpp index 48f5633..19d7e81 100644 --- a/tests/common-pg.cpp +++ b/tests/common-pg.cpp @@ -75,22 +75,7 @@ result::~result() { tempdb::tempdb() : m_conn(conn::connect("dbname=postgres")) { - result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE " - "spcname = 'tablespacetest'"); - - if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) || - (PQntuples(res->get()) != 1)) { - std::ostringstream out; - out << "The test needs a temporary tablespace to run in, but it does not " - << "exist. Please create the temporary tablespace. On Linux, you can " - << "do this by running:\n" - << " sudo mkdir -p /tmp/psql-tablespace\n" - << " sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n" - << " psql -c \"CREATE TABLESPACE tablespacetest LOCATION " - << "'/tmp/psql-tablespace'\" postgres\n"; - throw std::runtime_error(out.str()); - } - + result_ptr res = NULL; m_db_name = (boost::format("osm2pgsql-test-%1%-%2%") % getpid() % time(NULL)).str(); m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name); //tests can be run concurrently which means that this query can collide with other similar ones @@ -116,6 +101,24 @@ tempdb::tempdb() setup_extension(db, "hstore", NULL); } +void tempdb::check_tblspc() { + result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE " + "spcname = 'tablespacetest'"); + if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) || + (PQntuples(res->get()) != 1)) { + std::ostringstream out; + out << "The test needs a temporary tablespace to run in, but it does not " + << "exist. Please create the temporary tablespace. On Linux, you can " + << "do this by running:\n" + << " sudo mkdir -p /tmp/psql-tablespace\n" + << " sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n" + << " psql -c \"CREATE TABLESPACE tablespacetest LOCATION " + << "'/tmp/psql-tablespace'\" postgres\n"; + throw std::runtime_error(out.str()); + } + +} + tempdb::~tempdb() { if (m_conn) { m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name); diff --git a/tests/common-pg.hpp b/tests/common-pg.hpp index bb3cb08..faa2888 100644 --- a/tests/common-pg.hpp +++ b/tests/common-pg.hpp @@ -56,6 +56,8 @@ struct tempdb const std::string &conninfo() const; + void check_tblspc(); + private: void setup_extension(conn_ptr db, const std::string &extension, ...); diff --git a/tests/regression-test.py b/tests/regression-test.py index 617fd50..4ebcef6 100755 --- a/tests/regression-test.py +++ b/tests/regression-test.py @@ -168,11 +168,11 @@ sql_test_statements=[ ( 92, 'Basic line length', 'SELECT round(sum(ST_Length(way))) FROM planet_osm_roads;', 2032023), ( 93, 'Basic number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4228), ( 94, 'Basic number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2316), - ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 10897), + ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11131), ( 96, 'Basic number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9540), ( 97, 'Diff import number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4352), ( 98, 'Diff import number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2340), - ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11020), + ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11254), ( 100, 'Diff import number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9834), #**** Tests to check if inner polygon appears when outer tags change after initially identicall inner and outer way tags in a multi-polygon **** #**** These tests are currently broken and noted in trac ticket #2853 **** diff --git a/tests/test-middle-flat.cpp b/tests/test-middle-flat.cpp new file mode 100644 index 0000000..bb9e45d --- /dev/null +++ b/tests/test-middle-flat.cpp @@ -0,0 +1,112 @@ +#include <iostream> +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <cassert> +#include <sstream> +#include <stdexcept> +#include <memory> + +#include "osmtypes.hpp" +#include "output-null.hpp" +#include "options.hpp" +#include "middle-pgsql.hpp" + +#include <libpq-fe.h> +#include <sys/types.h> +#include <unistd.h> + +#include <boost/scoped_ptr.hpp> + +#include "tests/middle-tests.hpp" +#include "tests/common-pg.hpp" + +/* This is basically the same as test-middle-pgsql, but with flat nodes. */ + +void run_tests(options_t options, const std::string cache_type) { + options.append = 0; + options.create = 1; + options.flat_node_cache_enabled = true; + // flat nodes truncates the file each time it's started, so we can reuse the same file + options.flat_node_file = boost::optional<std::string>("tests/test_middle_flat.flat.nodes.bin"); + + { + middle_pgsql_t mid_pgsql; + output_null_t out_test(&mid_pgsql, options); + + mid_pgsql.start(&options); + + if (test_node_set(&mid_pgsql) != 0) { throw std::runtime_error("test_node_set failed."); } + + mid_pgsql.commit(); + mid_pgsql.stop(); + } + { + middle_pgsql_t mid_pgsql; + output_null_t out_test(&mid_pgsql, options); + + mid_pgsql.start(&options); + + if (test_nodes_comprehensive_set(&mid_pgsql) != 0) { throw std::runtime_error("test_nodes_comprehensive_set failed."); } + + mid_pgsql.commit(); + mid_pgsql.stop(); + } + /* This should work, but doesn't. More tests are needed that look at updates + without the complication of ways. + */ +/* { + middle_pgsql_t mid_pgsql; + output_null_t out_test(&mid_pgsql, options); + + mid_pgsql.start(&options); + mid_pgsql.commit(); + mid_pgsql.stop(); + // Switch to append mode because this tests updates + options.append = 1; + options.create = 0; + mid_pgsql.start(&options); + if (test_way_set(&mid_pgsql) != 0) { throw std::runtime_error("test_way_set failed."); } + + mid_pgsql.commit(); + mid_pgsql.stop(); + }*/ +} +int main(int argc, char *argv[]) { + boost::scoped_ptr<pg::tempdb> db; + + try { + db.reset(new pg::tempdb); + } catch (const std::exception &e) { + std::cerr << "Unable to setup database: " << e.what() << "\n"; + return 77; // <-- code to skip this test. + } + + try { + options_t options; + options.conninfo = db->conninfo().c_str(); + options.scale = 10000000; + options.cache = 1; + options.num_procs = 1; + options.prefix = "osm2pgsql_test"; + options.slim = 1; + + options.alloc_chunkwise = ALLOC_SPARSE | ALLOC_DENSE; // what you get with optimized + run_tests(options, "optimized"); + options.alloc_chunkwise = ALLOC_SPARSE; + run_tests(options, "sparse"); + + options.alloc_chunkwise = ALLOC_DENSE; + run_tests(options, "dense"); + + options.alloc_chunkwise = ALLOC_DENSE | ALLOC_DENSE_CHUNK; // what you get with chunk + run_tests(options, "chunk"); + } catch (const std::exception &e) { + std::cerr << "ERROR: " << e.what() << std::endl; + return 1; + } catch (...) { + std::cerr << "UNKNOWN ERROR" << std::endl; + return 1; + } + return 0; +} diff --git a/tests/test-output-multi-line.cpp b/tests/test-output-multi-line.cpp index cf53067..fe6b60b 100644 --- a/tests/test-output-multi-line.cpp +++ b/tests/test-output-multi-line.cpp @@ -61,8 +61,6 @@ int main(int argc, char *argv[]) { options_t options; options.conninfo = db->conninfo().c_str(); options.num_procs = 1; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; boost::shared_ptr<geometry_processor> processor = diff --git a/tests/test-output-multi-point-multi-table.cpp b/tests/test-output-multi-point-multi-table.cpp index d35f5bb..f76c147 100644 --- a/tests/test-output-multi-point-multi-table.cpp +++ b/tests/test-output-multi-point-multi-table.cpp @@ -67,8 +67,6 @@ int main(int argc, char *argv[]) { options.conninfo = db->conninfo().c_str(); options.num_procs = 1; options.prefix = "osm2pgsql_test"; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; export_list columns; diff --git a/tests/test-output-multi-point.cpp b/tests/test-output-multi-point.cpp index 36e102a..fe29481 100644 --- a/tests/test-output-multi-point.cpp +++ b/tests/test-output-multi-point.cpp @@ -62,8 +62,6 @@ int main(int argc, char *argv[]) { options.conninfo = db->conninfo().c_str(); options.num_procs = 1; options.prefix = "osm2pgsql_test"; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; boost::shared_ptr<geometry_processor> processor = diff --git a/tests/test-output-multi-polygon.cpp b/tests/test-output-multi-polygon.cpp index 38be794..98370c6 100644 --- a/tests/test-output-multi-polygon.cpp +++ b/tests/test-output-multi-polygon.cpp @@ -62,8 +62,6 @@ int main(int argc, char *argv[]) { options.conninfo = db->conninfo().c_str(); options.num_procs = 1; options.prefix = "osm2pgsql_test"; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; boost::shared_ptr<geometry_processor> processor = geometry_processor::create("polygon", &options); diff --git a/tests/test-output-pgsql-tablespace.cpp b/tests/test-output-pgsql-tablespace.cpp new file mode 100755 index 0000000..3f75299 --- /dev/null +++ b/tests/test-output-pgsql-tablespace.cpp @@ -0,0 +1,146 @@ +#include <iostream> +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <cassert> +#include <sstream> +#include <stdexcept> +#include <memory> + +#include "osmtypes.hpp" +#include "osmdata.hpp" +#include "output-pgsql.hpp" +#include "options.hpp" +#include "middle-pgsql.hpp" +#include "middle-ram.hpp" +#include "taginfo_impl.hpp" +#include "parse.hpp" + +#include <libpq-fe.h> +#include <sys/types.h> +#include <unistd.h> + +#include <boost/scoped_ptr.hpp> +#include <boost/lexical_cast.hpp> + +#include "tests/middle-tests.hpp" +#include "tests/common-pg.hpp" + +namespace { + +struct skip_test : public std::exception { + const char *what() { return "Test skipped."; } +}; + +void run_test(const char* test_name, void (*testfunc)()) { + try { + fprintf(stderr, "%s\n", test_name); + testfunc(); + + } catch (const skip_test &) { + exit(77); // <-- code to skip this test. + + } catch (const std::exception& e) { + fprintf(stderr, "%s\n", e.what()); + fprintf(stderr, "FAIL\n"); + exit(EXIT_FAILURE); + } + + fprintf(stderr, "PASS\n"); +} +#define RUN_TEST(x) run_test(#x, &(x)) + +void check_count(pg::conn_ptr &conn, int expected, const std::string &query) { + pg::result_ptr res = conn->exec(query); + + int ntuples = PQntuples(res->get()); + if (ntuples != 1) { + throw std::runtime_error((boost::format("Expected only one tuple from a query " + "to check COUNT(*), but got %1%. Query " + "was: %2%.") + % ntuples % query).str()); + } + + std::string numstr = PQgetvalue(res->get(), 0, 0); + int count = boost::lexical_cast<int>(numstr); + + if (count != expected) { + throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running " + "query: %3%.") + % expected % count % query).str()); + } +} + +void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) { + std::string query = (boost::format("select count(*) from pg_catalog.pg_class " + "where relname = '%1%'") + % table_name).str(); + + check_count(test_conn, 1, query); +} + +// "simple" test modeled on the basic regression test from +// the python script. this is just to check everything is +// working as expected before we start the complex stuff. +void test_regression_simple() { + boost::scoped_ptr<pg::tempdb> db; + + try { + db.reset(new pg::tempdb); + db->check_tblspc(); // Unlike others, these tests require a test tablespace + } catch (const std::exception &e) { + std::cerr << "Unable to setup database: " << e.what() << "\n"; + throw skip_test(); + } + + std::string proc_name("test-output-pgsql"), input_file("-"); + char *argv[] = { &proc_name[0], &input_file[0], NULL }; + + boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t()); + options_t options = options_t::parse(2, argv); + options.conninfo = db->conninfo().c_str(); + options.num_procs = 1; + options.prefix = "osm2pgsql_test"; + options.slim = 1; + options.style = "default.style"; + + options.tblsslim_index = "tablespacetest"; + options.tblsslim_data = "tablespacetest"; + + boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options)); + + osmdata_t osmdata(mid_pgsql, out_test); + + boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection)); + + osmdata.start(); + + if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) { + throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'."); + } + + parser.reset(NULL); + + osmdata.stop(); + + // start a new connection to run tests on + pg::conn_ptr test_conn = pg::conn::connect(db->conninfo()); + + assert_has_table(test_conn, "osm2pgsql_test_point"); + assert_has_table(test_conn, "osm2pgsql_test_line"); + assert_has_table(test_conn, "osm2pgsql_test_polygon"); + assert_has_table(test_conn, "osm2pgsql_test_roads"); + + check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point"); + check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line"); + check_count(test_conn, 375, "SELECT count(*) FROM osm2pgsql_test_roads"); + check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon"); +} + +} // anonymous namespace + +int main(int argc, char *argv[]) { + RUN_TEST(test_regression_simple); + + return 0; +} diff --git a/tests/test-output-pgsql-z_order.cpp b/tests/test-output-pgsql-z_order.cpp new file mode 100644 index 0000000..c940f44 --- /dev/null +++ b/tests/test-output-pgsql-z_order.cpp @@ -0,0 +1,186 @@ +#include <iostream> +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <cassert> +#include <sstream> +#include <stdexcept> +#include <memory> + +#include "osmtypes.hpp" +#include "osmdata.hpp" +#include "output-pgsql.hpp" +#include "options.hpp" +#include "middle-pgsql.hpp" +#include "middle-ram.hpp" +#include "taginfo_impl.hpp" +#include "parse.hpp" + +#include <libpq-fe.h> +#include <sys/types.h> +#include <unistd.h> + +#include <boost/scoped_ptr.hpp> +#include <boost/lexical_cast.hpp> + +#include "tests/middle-tests.hpp" +#include "tests/common-pg.hpp" + +namespace { + +struct skip_test : public std::exception { + const char *what() { return "Test skipped."; } +}; + +void run_test(const char* test_name, void (*testfunc)()) { + try { + fprintf(stderr, "%s\n", test_name); + testfunc(); + + } catch (const skip_test &) { + exit(77); // <-- code to skip this test. + + } catch (const std::exception& e) { + fprintf(stderr, "%s\n", e.what()); + fprintf(stderr, "FAIL\n"); + exit(EXIT_FAILURE); + } + + fprintf(stderr, "PASS\n"); +} +#define RUN_TEST(x) run_test(#x, &(x)) + +void check_string(pg::conn_ptr &conn, std::string expected, const std::string &query) { + pg::result_ptr res = conn->exec(query); + + int ntuples = PQntuples(res->get()); + if (ntuples != 1) { + throw std::runtime_error((boost::format("Expected only one tuple from a query " + "to check a string, but got %1%. Query " + "was: %2%.") + % ntuples % query).str()); + } + + std::string actual = PQgetvalue(res->get(), 0, 0); + + if (actual != expected) { + throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running " + "query: %3%.") + % expected % actual % query).str()); + } +} + + +void check_count(pg::conn_ptr &conn, int expected, const std::string &query) { + pg::result_ptr res = conn->exec(query); + + int ntuples = PQntuples(res->get()); + if (ntuples != 1) { + throw std::runtime_error((boost::format("Expected only one tuple from a query " + "to check COUNT(*), but got %1%. Query " + "was: %2%.") + % ntuples % query).str()); + } + + std::string numstr = PQgetvalue(res->get(), 0, 0); + int count = boost::lexical_cast<int>(numstr); + + if (count != expected) { + throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running " + "query: %3%.") + % expected % count % query).str()); + } +} + +void check_number(pg::conn_ptr &conn, double expected, const std::string &query) { + pg::result_ptr res = conn->exec(query); + + int ntuples = PQntuples(res->get()); + if (ntuples != 1) { + throw std::runtime_error((boost::format("Expected only one tuple from a query, " + " but got %1%. Query was: %2%.") + % ntuples % query).str()); + } + + std::string numstr = PQgetvalue(res->get(), 0, 0); + double num = boost::lexical_cast<double>(numstr); + + // floating point isn't exact, so allow a 0.01% difference + if ((num > 1.0001*expected) || (num < 0.9999*expected)) { + throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running " + "query: %3%.") + % expected % num % query).str()); + } +} + +void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) { + std::string query = (boost::format("select count(*) from pg_catalog.pg_class " + "where relname = '%1%'") + % table_name).str(); + + check_count(test_conn, 1, query); +} + +// "simple" test modeled on the basic regression test from +// the python script. this is just to check everything is +// working as expected before we start the complex stuff. +void test_z_order() { + boost::scoped_ptr<pg::tempdb> db; + + try { + db.reset(new pg::tempdb); + } catch (const std::exception &e) { + std::cerr << "Unable to setup database: " << e.what() << "\n"; + throw skip_test(); + } + + std::string proc_name("test-output-pgsql"), input_file("-"); + char *argv[] = { &proc_name[0], &input_file[0], NULL }; + + boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t()); + options_t options = options_t::parse(2, argv); + options.conninfo = db->conninfo().c_str(); + options.num_procs = 1; + options.prefix = "osm2pgsql_test"; + options.style = "default.style"; + + boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options)); + + osmdata_t osmdata(mid_pgsql, out_test); + + boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection)); + + osmdata.start(); + + if (parser->streamFile("libxml2", "tests/test_output_pgsql_z_order.osm", options.sanitize, &osmdata) != 0) { + throw std::runtime_error("Unable to read input file `tests/test_output_pgsql_z_order.osm'."); + } + + parser.reset(NULL); + + osmdata.stop(); + + // start a new connection to run tests on + pg::conn_ptr test_conn = pg::conn::connect(db->conninfo()); + + assert_has_table(test_conn, "osm2pgsql_test_point"); + assert_has_table(test_conn, "osm2pgsql_test_line"); + assert_has_table(test_conn, "osm2pgsql_test_polygon"); + assert_has_table(test_conn, "osm2pgsql_test_roads"); + + check_string(test_conn, "motorway", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 0"); + check_string(test_conn, "trunk", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 1"); + check_string(test_conn, "primary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 2"); + check_string(test_conn, "secondary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 3"); + check_string(test_conn, "tertiary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 4"); + + check_string(test_conn, "residential", "SELECT highway FROM osm2pgsql_test_line ORDER BY z_order DESC LIMIT 1 OFFSET 0"); +} + +} // anonymous namespace + +int main(int argc, char *argv[]) { + RUN_TEST(test_z_order); + + return 0; +} diff --git a/tests/test-output-pgsql.cpp b/tests/test-output-pgsql.cpp index 42f490a..b7a4053 100644 --- a/tests/test-output-pgsql.cpp +++ b/tests/test-output-pgsql.cpp @@ -71,6 +71,27 @@ void check_count(pg::conn_ptr &conn, int expected, const std::string &query) { } } +void check_number(pg::conn_ptr &conn, double expected, const std::string &query) { + pg::result_ptr res = conn->exec(query); + + int ntuples = PQntuples(res->get()); + if (ntuples != 1) { + throw std::runtime_error((boost::format("Expected only one tuple from a query, " + " but got %1%. Query was: %2%.") + % ntuples % query).str()); + } + + std::string numstr = PQgetvalue(res->get(), 0, 0); + double num = boost::lexical_cast<double>(numstr); + + // floating point isn't exact, so allow a 0.01% difference + if ((num > 1.0001*expected) || (num < 0.9999*expected)) { + throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running " + "query: %3%.") + % expected % num % query).str()); + } +} + void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) { std::string query = (boost::format("select count(*) from pg_catalog.pg_class " "where relname = '%1%'") @@ -100,8 +121,6 @@ void test_regression_simple() { options.conninfo = db->conninfo().c_str(); options.num_procs = 1; options.prefix = "osm2pgsql_test"; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; options.style = "default.style"; @@ -133,8 +152,85 @@ void test_regression_simple() { check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line"); check_count(test_conn, 375, "SELECT count(*) FROM osm2pgsql_test_roads"); check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon"); + + // Check size of lines + check_number(test_conn, 1696.04, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682"); + check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682"); + + check_number(test_conn, 311.21, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + check_number(test_conn, 311.21, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + check_number(test_conn, 143.81, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + + // Check a point's location + check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=900913;POINT(1062645.12 5972593.4)'::geometry, 0.1)"); +} + +void test_latlong() { + boost::scoped_ptr<pg::tempdb> db; + + try { + db.reset(new pg::tempdb); + } catch (const std::exception &e) { + std::cerr << "Unable to setup database: " << e.what() << "\n"; + throw skip_test(); + } + + std::string proc_name("test-output-pgsql"), input_file("-"); + char *argv[] = { &proc_name[0], &input_file[0], NULL }; + + boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t()); + options_t options = options_t::parse(2, argv); + options.conninfo = db->conninfo().c_str(); + options.num_procs = 1; + options.prefix = "osm2pgsql_test"; + options.slim = 1; + options.style = "default.style"; + + options.projection.reset(new reprojection(PROJ_LATLONG)); + options.scale = (options.projection->get_proj_id() == PROJ_LATLONG) ? 10000000 : 100; + + boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options)); + + osmdata_t osmdata(mid_pgsql, out_test); + + boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection)); + + osmdata.start(); + + if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) { + throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'."); + } + + parser.reset(NULL); + + osmdata.stop(); + + // start a new connection to run tests on + pg::conn_ptr test_conn = pg::conn::connect(db->conninfo()); + + assert_has_table(test_conn, "osm2pgsql_test_point"); + assert_has_table(test_conn, "osm2pgsql_test_line"); + assert_has_table(test_conn, "osm2pgsql_test_polygon"); + assert_has_table(test_conn, "osm2pgsql_test_roads"); + + check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point"); + check_count(test_conn, 3298, "SELECT count(*) FROM osm2pgsql_test_line"); + check_count(test_conn, 374, "SELECT count(*) FROM osm2pgsql_test_roads"); + check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon"); + + // Check size of lines + check_number(test_conn, 0.0105343, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682"); + check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682"); + + check_number(test_conn, 1.70718e-08, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + check_number(test_conn, 1.70718e-08, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + check_number(test_conn, 143.845, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342"); + + // Check a point's location + check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=4326;POINT(9.5459035 47.1866494)'::geometry, 0.00001)"); } + void test_area_way_simple() { boost::scoped_ptr<pg::tempdb> db; @@ -259,8 +355,6 @@ void test_clone() { options.conninfo = db->conninfo().c_str(); options.num_procs = 1; options.prefix = "osm2pgsql_test"; - options.tblsslim_index = "tablespacetest"; - options.tblsslim_data = "tablespacetest"; options.slim = 1; options.style = "default.style"; @@ -302,6 +396,7 @@ void test_clone() { int main(int argc, char *argv[]) { RUN_TEST(test_regression_simple); + RUN_TEST(test_latlong); RUN_TEST(test_clone); RUN_TEST(test_area_way_simple); RUN_TEST(test_route_rel); diff --git a/tests/test_output_pgsql_z_order.osm b/tests/test_output_pgsql_z_order.osm new file mode 100644 index 0000000..76ddd9a --- /dev/null +++ b/tests/test_output_pgsql_z_order.osm @@ -0,0 +1,75 @@ +<?xml version='1.0' encoding='UTF-8'?> +<osm version='0.6' generator='hand'> + <node id='1' version='1' visible='true' lat='49' lon='-122.5'> + <tag k='highway' v='bus_stop' /> + </node> + <node id='2' version='1' visible='true' lat='49.005' lon='-122.51'> + <tag k='highway' v='bus_stop' /> + </node> + <node id='3' version='1' visible='true' lat='49.01' lon='-122.5' /> + <!-- yes, all the ways have the same linestring --> + <way id='1' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='motorway' /> + </way> + <way id='2' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='motorway_link' /> + </way> + <way id='3' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='trunk' /> + </way> + <way id='4' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='trunk_link' /> + </way> + <way id='5' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='primary' /> + </way> + <way id='6' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='primary_link' /> + </way> + <way id='7' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='secondary' /> + </way> + <way id='8' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='secondary_link' /> + </way> + <way id='9' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='tertiary' /> + </way> + <way id='10' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='tertiary_link' /> + </way> + <way id='11' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <tag k='highway' v='residential' /> + <tag k='layer' v='5' /> + </way> + <!-- put *something* in the polygon table --> + <way id='1000' version='1' visible='true'> + <nd ref='1' /> + <nd ref='2' /> + <nd ref='3' /> + <nd ref='1' /> + <tag k='building' v='yes' /> + </way> +</osm> -- Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-grass/osm2pgsql.git _______________________________________________ Pkg-grass-devel mailing list Pkg-grass-devel@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-grass-devel