Re: [Taginfo-dev] Problem updating to latest taginfo

2021-01-05 Thread Sarah Hoffmann
On Tue, Jan 05, 2021 at 11:16:07AM +0100, Jochen Topf wrote:
> Hi Shaun,
> 
> On Tue, Jan 05, 2021 at 10:01:03AM +, Shaun McDonald via Taginfo-dev 
> wrote:
> > I’ll do another review of the config to see if there’s anything else that 
> > needs to be updated. I think the main thing is enabling the chronology 
> > functionality, and to do that needs a regional history file, which I don’t 
> > know how to produce yet. Any pointers appreciated.
> 
> You can get a regional history file by downloading the planet history
> file and using "osmium extract" on it. You can get the poly file for
> your region from the Geofabrik download server.
> 
> I have not tried this, but you should then be able to update that
> history file using the diffs from Geofabrik. The pyosmium-up-to-date
> command from pyosmium can update history files (you need the newest
> version).

The Geofabrik download server also has history extracts for the
regions. They are available on the internal server only.

For example: 
https://osm-internal.download.geofabrik.de/europe/ireland-and-northern-ireland.html
Second entry under 'other formats'.

Sarah

___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev


Re: [OSM-dev] Automatic generation of short_name / alt_name where missing and obvious

2020-11-22 Thread Sarah Hoffmann
On Sat, Nov 21, 2020 at 07:15:38PM +0100, Roland Olbricht via dev wrote:
> > > Is there some existing code/library/tool for generating obvious
> > > short_name / alt_name from other tagged data?
> > 
> > Specifically for shortened names, this list of common abbreviations
> > could help: https://wiki.osm.org/Name_finder:Abbreviations
> 
> Thank you for the pointer. At least for German (and for French as far as
> I can judge) please take it with a grain of salt:
> - The by far fast common practices to shorten names are to resort to
> KFZ-Kennzeichen (district codes), i.e. "K-Süd" is the short_name for
> "Köln-Süd"
> - In 95% of Germany (everywhere except Berlin), the abbreviaton of
> "Bahnhof" is "Bf"
> - the suffix "straße" resp. "Straße" is shortened to "str." resp.
> "Str.", everything else is uncommon
> - French street names are shortened by entirely removing the generic
> parts, i.e. "Place Victor Hugo" has short_name "Victor Hugo"

... unless they are shortened by removing the first name.
Read https://github.com/osm-search/Nominatim/issues/679 for a
couple of fun examples.

Three more libraries that collect lists of abbreviations:

https://github.com/openvenues/libpostal/tree/master/resources/dictionaries
https://github.com/OpenCageData/address-formatting/tree/master/conf/abbreviations
https://github.com/mapbox/geocoder-abbreviations

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Comparing quality of different geocoders?

2020-08-26 Thread Sarah Hoffmann
Hi,

On Wed, Aug 26, 2020 at 09:11:44AM +0200, Mateusz Konieczny via dev wrote:
> Is anyone aware about some test suite comparing different geocoders?
> 
> Something with query + expected result pairs, and listing where Nominatim / 
> Photon / etc succeeded/failed?
> 
> I want something like that, tried to find one and failed.
> 
> I want to ask whatever I missed something like that before I will make it.

There are a couple of test suits out there. The two that come
immediately to mind are:
https://github.com/geocoders/geocoder-tester
https://github.com/pelias/fuzzy-tester

The issue with all these is that you have to define what you
actually want to compare. Depending on what kind of queries you
run and how you define the results you get, you'll get very different
results.

Sarah


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM Carto stylesheet vs Release osm2pgsql 1.3.0

2020-07-29 Thread Sarah Hoffmann
On Wed, Jul 29, 2020 at 02:59:24PM +0200, Christoph Hormann wrote:
> On Wednesday 29 July 2020, Lynn W. Deffenbaugh (Mr) wrote:
> > >
> > >   * The pgsql output now looks for lua script relative to the
> > >
> > > |style.json| file.
> > >
> > > This is a breaking change. Users might have to change the file
> > > names of
> > > their lua scripts in the style files.
> >
> > Does anyone know if changes need to be made to the OpenStreetMap
> > Carto stylesheet or is it already compatible with this breaking
> > change?
> 
> OSM-Carto has no style.json file, you specify the lua script directly at 
> the command line - i think the quoted change applies to the multi/flex 
> backends only, though the documentation is not very clear in the 
> terminology used ('pgsql output' vs. 'pgsql backend').

Indeed, that was a minor error in the release notes. I fixed that.

Filenames given on the command line are relative to the current
path. Only file names that appear in style files are interpreted
relative to the style file. The pgsql output used by Carto does
not have the latter. So: nothing changes.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] osm2pgsql 1.3.0 released

2020-07-28 Thread Sarah Hoffmann
Hi,

we are happy to announce a new release 1.3.0 of osm2pgsql.

This release sees the addition of the (still experimental) new
flex output. This gives more flexibilty how output tables in
Postgresql are structured and filled. It also introduces the
possibility to forward information from relations to ways.
Read more at
https://blog.jochentopf.com/2020-07-28-osm2pgsql-release.html
and give it a try.

Apart from the flex output the code has been improved in lots of other
places. For the full release notes see
https://github.com/openstreetmap/osm2pgsql/releases/tag/1.3.0

Special thanks to Jochen Topf, who has contributed most of the code
for this release.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim 3.5.1 released

2020-06-29 Thread Sarah Hoffmann
Hi all,

we have just released version 3.5.1 of Nominatim.

This bug-fix release fixes two important issues with osm2pgsql:

* osm2pgsql might get stuck during updates when running with Postgresql 12
* osm2pgsql might hang when processing extremely complex multipolygons

All users should update to this latest version when they run regular updates
on their database or plan to import new data that includes North America. 
Nominatim 3.5.0 installations can be updated directly without changes to the 
database. For upgrading from older versions please consult the migration guide.

For more information visit https://nominatim.org

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] New osm2pgsql release 1.2.2

2020-06-28 Thread Sarah Hoffmann
Dear all,

we have just released a new version 1.2.2 of osm2pgsql.

This is a bugfix release which updates the bundled libosmium only.
It fixes yesterday's issue where osm2pgsql updates stalled on a
large multi-polygon relation.

All users are strongly recommended to update to this newest release.
Data containing the relation is not importable or updatable by
older versions of osm2pgsql.

The release can be downloaded from github:
https://github.com/openstreetmap/osm2pgsql/archive/1.2.2.tar.gz
For Windows versions, see:
https://lonvia.dev.openstreetmap.org/osm2pgsql-winbuild/releases/

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim: security bug fix release

2020-05-04 Thread Sarah Hoffmann
Hi all,

A few days ago we have been informed about a security vulnerability in the
Nominatim API. Today we have released updates for all affected Nominatim
versions.

Today we have released new versions 3.4.2, 3.3.1 and 3.2.1 of Nominatim.
If you have your own installation of Nominatim, you should update as soon
as possible.

What is the problem?

  The /details endpoint fails to properly sanitize user input and uses it
  as is in an SQL query. This allows an attacker to inject arbitrary SQL
  code including querying and updating the database.

Which versions are affected?

  The code was added to Nominatim in April 2018. All releases since 3.2
  are affected. The bug has been fixed in 3.4.2, 3.3.1 and 3.2.1.

How is my installation affected?

  If you have followed the standard installation instructions, then the
  /details endpoint is available by default. The standard installation also
  adds a special user for the webserver which has only minimal read rights
  on the database. If you have not changed the rights, then the vulnerability
  can only be used to query the database.

How should I fix it?

  If you don't need the details API, then you can simply delete the file
  `website/details.php` to remove the endpoint. Otherwise, you should install
  the appropriate update for your version. No changes to the database are
  necessary. Simply download and build the new version, copy over your
  `settings/local.php` file and point your webserver to the new version.

A big thank you to @bladeswords for finding and reporting this.

Kind regards

Sarah


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Issue with way id 742619958 / changeset 76711047 from 2019-11-06T15:32:03Z (Invalid layer value)

2019-11-20 Thread Sarah Hoffmann
On Fri, Nov 08, 2019 at 05:04:34PM +0100, Bart Smienk wrote:
> Hi,
> 
> I've found out my OSM Tile Server has stopped updating and osm2pgsql (v1.2)
> gives off the following error:
> 
> DB writer thread failed due to ERROR: result COPY_END for planet_osm_line
> failed: ERROR:  invalid input syntax for integer: "-"
> CONTEXT:  COPY planet_osm_line, line 1437, column layer: "-"
> 
> Looking through its changefile, the following entry can be found:
> 
>  uid="978786" user="padvinder" changeset="76711047">
>   
>   
>   
>   
>   
> 
> 
> This seems to be malformed, as the layer should've never been "-", as this
> is not a valid integer.
> You can see this for yourself at:
> https://planet.openstreetmap.org/replication/minute/003/747/729.osc.gz
> Also more information on the layer key is over here:
> https://wiki.openstreetmap.org/wiki/Key:layer
> 
> Not sure where to make the actual bug report, so I'm mailing the dev
> mailgroup, does anyone of you know where this issue should be posted so this
> issue won't present itself again in future replication files?

Looks like a potential bug in osm2pgsql's int parsing. I've opened an issue:
https://github.com/openstreetmap/osm2pgsql/issues/988

Please follow there for further updates.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] QA, check for tag change

2019-10-03 Thread Sarah Hoffmann
On Wed, Oct 02, 2019 at 09:30:26PM +0200, Frederik Ramm wrote:
> Hi,
> 
> On 10/2/19 20:36, yves wrote:
> > What would be a cheap way to find ways that once were highway=* and are
> > now a piste:type=nordic but no more a highway=* ?
> 
> Define "cheap" ;)
> 
> I usually tackle issue like this with a contraption of history file,
> osmium, and a script in a programming language of choice (for easy cases
> even shell script works).
> 
> The first step is to have osmium convert the history file into "OPL"
> format which gives you one ascii text line for every version of an
> object, neatly sorted by type, ID, and version:

If I may add to that: I would first filter the history file for
the intersting data (i.e. ways with piste:type=nordic) with 'osmium tags-filter'
before doing anything else. The resulting file should be much more
managable.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] PostgreSQL 11 - osm2pgsql performance problems during --append?

2019-09-13 Thread Sarah Hoffmann
Hi,

On Fri, Sep 13, 2019 at 01:07:39PM +0200, Michael Kussmaul wrote:
> local network: constant 17.8Mb in and 17.8Mb out. It looks like those are 
> postgres UDP packets sent/received (according to lsof), googling it seems 
> those are from the stats collector. Perhaps this is my problem - I will 
> disable stats on the next run.
> 
> I then have mostly two postgres processes consuming CPU
> 19% CPU on "main: gis planet [local] SELECT" the queries on the DB
> 13% CPU on "/usr/lib/postgresql/11/bin/postgres" the binary itself
> 1-4% CPU on osm2pgsql

Please have a look what exactly the SELECT is doing. My guess would
be that you have lost an index during the upgrade to PG11 and it is
now sequentially scanning the tables.

Check for these indexes:

planet_osm_nodes : planet_osm_nodes_pkey
planet_osm_ways : planet_osm_ways_pkey, planet_osm_ways_nodes

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] osm2pgsql 1.0.0 released

2019-08-29 Thread Sarah Hoffmann
Hi all,

we are happy to announce a new release of osm2pgsql. This release drops
support for old-style multipolygon, which we celebrate by making a big
version jump to osm2pgsql 1.0.0.

Since the last release, the processing pipeline of osm2pgsql has received
a major internal overhaul. Imports are now entirely done in the first
processing stage. The second, much slower processing stage is only needed
when updating existing databases. Data is entirely streamed into the database
using COPY, which reduces the number of connections needed in the database.

Other major changes include

- better error handling in Lua backend
- process all OSM objects again when extra attributes are requested
- enable running tests in pg_virtualenv
- add support for configurable Gazetteer style
- allow to disable RAM node cache with -C 0

Starting with this release you can find the unofficial release builds for
Windows also on the OpenStreetMap dev server:
https://lonvia.dev.openstreetmap.org/osm2pgsql-winbuild/releases/

Thanks to everyone who contributed to this release.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Python XML Parser on Windows

2019-08-07 Thread Sarah Hoffmann
On Wed, Aug 07, 2019 at 06:39:14AM -0700, Spencer Gardner wrote:
> Thanks. I'm aware of PyOsmium, which appears to work on Windows but not
> without installing a number of additional packages related to the
> libosmium. I may go that route in the absence of easier alternatives, but
> installation of libosmium doesn't seem very straightforward on Windows. Or
> perhaps the problem is the documentation doesn't provide much information
> on how to install on Windows short of building from source, which is a
> nonstarter for most users. What I'm really hoping for is an isolated Python
> solution that can be installed via a simple pip command.

We do provide precomiled binaries for Windows (64bit), so you
should be able to just 'pip install osmium' without building libosmium
yourself.

I admit that this is not well tested because we don't have a maintainer
with a Windows development platform (and experience with it). If you run
into trouble, feel free to file an issue and we see what we can do.

Sarah


> 
> Have I misunderstood something about PyOsmium and libosmium?
> 
> On Wed, Aug 7, 2019 at 1:03 AM Jochen Topf  wrote:
> 
> > On Wed, Aug 07, 2019 at 08:30:07AM +0100, Sarah Hoffmann wrote:
> > > On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> > > > Hi!
> > > >
> > > > PyOsmium https://osmcode.org/pyosmium/ should work on Windows.
> > >
> > > Just stay away from bz2 compressed xml. That's known to be
> > > broken on Windows. Uncompressed xml should be fine though.
> >
> > Or, even better, use the PBF file format which should be much faster
> > than XML.
> >
> > Jochen
> > --
> > Jochen Topf  joc...@remote.org  https://www.jochentopf.com/
> > +49-351-31778688
> >

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Python XML Parser on Windows

2019-08-07 Thread Sarah Hoffmann
On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> Hi!
> 
> PyOsmium https://osmcode.org/pyosmium/ should work on Windows.

Just stay away from bz2 compressed xml. That's known to be
broken on Windows. Uncompressed xml should be fine though.

Sarah

> On Tue, Aug 06, 2019 at 09:20:39PM -0700, Spencer Gardner wrote:
> > Date: Tue, 6 Aug 2019 21:20:39 -0700
> > From: Spencer Gardner 
> > To: dev@openstreetmap.org
> > Subject: [OSM-dev] Python XML Parser on Windows
> > 
> > I'm sort of resurrecting an old thread I started last year. I find
> > Martijn's Overpass Python library to be very useful for small queries but I
> > frequently run into usage limits. I'd like to query an XML download
> > directly to save bandwidth and reduce consumption of Overpass resources,
> > but I can't find any suitable Python libraries that are WIndows-ready. Is
> > there something I'm missing? Has anyone else found a solution that works?
> > 
> > By way of example, Geoff Boeing's excellent osmnx package appears to use a
> > home-grown OSM parser due to lack of a cross-platform option.
> > https://github.com/gboeing/osmnx/blob/eebbca0ed1d59a6dfe07c90493d54c0beab45145/osmnx/utils.py#L1181
> > 
> > Thanks for any help
> 
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
> 
> 
> -- 
> Jochen Topf  joc...@remote.org  https://www.jochentopf.com/  +49-351-31778688
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim 3.3.0 released

2019-05-02 Thread Sarah Hoffmann
Hi,

we are happy to announce the new release 3.3.0 of Nominatim.

This latest release makes Nominatim more configurable in custom installations.
There is a new reverse-only mode for imports where forward search is not
needed. In addition it is now possible to configure which OSM tags are taken
into account by Nominatim. This allows to describe exactly which types of
objects should be found in your instance. There are a number of example
configurations for frequent use cases which you might use as a starting
point. For more information read the installation documentation at
http://nominatim.org/release-docs/latest/

As usual there are also a number of bug fixes and minor improvements.
Details can be found in the ChangeLog file.

If you want to update from 3.x.x, there is a database migration path described
in the migration guide. Migrating from 3.1.0 can be done without problems.
However, when updating from 3.0.x, a full database reimport is strongly
recommended. Updating from older releases is only possible with
a database reimport.

This will be the last release to support PHP 5.x and PostgreSQL < 9.3.

Thank you to all who have contributed and in particular to the first
time contributor Thomas Barris.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] GSOC 2019 project suggestions feedback

2019-03-28 Thread Sarah Hoffmann
Hi Daniel,

thanks for your interest in the OpenStreetMap GSoC.

From the projects that you have listed, we have moved the one about
3D mapping features to an 'incubator' state meaning that the idea still
needs more work to be shaped into a full GSoC project proposal and that
you should actively search for a mentor.

The other three suggestions are equally useful and you should try to find
the project that matches best your skills and interest. The projects are
quite different each.

The iD projects are good if you like Javascript and web development.
I can't give you details about the project itself. Brian Housel can
talk about the details.

The Nominatim Wikidata project is more focused on databases and data
processing. Essentially it is about extracting helpful information
from the wikidata database and integrating it into the Nominatim
database and using it to improve the search results. Find out what
exactly is 'helpful' is a big part of the project. So you should like
data analysis and writing scripts for processing large amout of data.

The search suggestion project has a bit of both. photon is an
elastic search based frontend for Nominatim, which should be tuned
for the specific purpose of providing suggestions for the search box
on openstreetmap.org. Here som Java skills are essential and it
would help if you have heard of Elastic Search previously. Some
basic web programming skills for the integration into the website
are also useful.

I hope that helps a bit better to decide what you think suits you
best. For the Nominatim projects I'm happy to answer further
questions.

Kind regards

Sarah


On Tue, Mar 26, 2019 at 10:01:27PM -0400, Daniel Wu wrote:
> Hi everyone,
> 
> I am interested in participating in Google Summer of Code this year and
> working on a project with OpenStreetMap. I would like to either work with
> the iD editor or with Nominatim. I looked at the project suggestions for
> this year and there are some that interest me and I would like to know what
> others think about these projects.
> 
> For Nominatim, one project is to add Wikidata or other data to improve the
> importance ranking for Nominatim and another is to add search suggestions
> for OSM while you type. (
> https://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2019/Project_Ideas#Nominatim)For
> iD, I am interested in one project suggestion to integrate a task manager
> into iD and another project suggestion to create a new panel in iD to
> select select building color, roof color and roof shape and orientation. (
> https://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2019/Project_Ideas#iD_Editor
> ).
> 
> I would like to know which project you think would be most useful and any
> other opinions you have related to these projects.
> 
> Thanks,
> 
> Daniel
> -- 
> Daniel Wu
> University of Michigan | 2020
> B.S. Computer Science
> B.S. Ecology and Evolutionary Biology
> danie...@umich.edu
> 248-207-5214

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Stalls on updates with libosmium-based software (osm2pgsql, Nominatim, osmium)

2019-03-27 Thread Sarah Hoffmann
Hi,

this morning a rather large multipolygon relation (9436485) has triggered
a bug in the polygon-creation code in libosmium. The code will end up
in an endless loop when processing the relation. All versions of
libosmium v2.12.2 or larger are affected. We are still busy fixing
the issue.

If you are a consumer of minutely or hourly diffs using libosmium-based
software (osmium, osm2pgsql >= 0.94, Nominatim, etc.) you are likely
affected and will see that updates have stalled. The relation has been
deleted in the meantime. So the recommended workaround is to consume
a combined diff of at least 3 hours to circumvent importing the relation.

Stop any running instance importing a diff with the relation and restart
with a larger diff.

When downloading updates with the pyosmium tools use -s 200 or larger.
When using osmosis, set maxInterval in your configuration.txt to 10800
or larger. For Nominatim set the option CONST_Replication_Max_Diff_size
in your settings/local.php to 200 or larger.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Sarah Hoffmann
Hi,

On Thu, Dec 20, 2018 at 08:54:34PM -0800, Spencer Gardner wrote:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

If you are looking for something purely Python, then you can combine
pyosmium for reading OSM data with SQLAlchemy for writing it into
the database. This gives you maximum flexibility for the database
schema.

For an example on how to do it:
https://github.com/waymarkedtrails/osgende/blob/master/tools/osgende-import

This tool only imports the raw OSM data. If you want preprocessed
geometries then you would need to have a look at pyosmiums shapely
example. 

This is quite a bit slower than the Java and C++ importers mentioned in this
thread but that would only be relevant if you do regular full planet
imports.

Regarding Windows compatibility: pyosmium does have a native library part.
We do create pre-compiled versions for installation via pip but only for
a handful of python versions and so far I had neither positive nor negative
feedback if that actually works as intended.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] iD news - 2.12.0 released 

2018-12-10 Thread Sarah Hoffmann
On Mon, Dec 10, 2018 at 09:54:23AM -0500, Bryan Housel wrote:
> Sure, I can describe the logic in more detail.  
> 
> When I talk about “fields”, I mean the fields near the top of iD’s sidebar.  
> Users can still edit all the raw tags in the tag editor that appears below 
> the fields.  
> 
> The purpose of this feature is not really to prevent someone who knows what 
> they are doing from changing a tag value, but more to hint to users 
> (especially newbies) “hey maybe you should not change that”.  
> 
> The protection should discourage people from changing the Name field just for 
> fun, but also make sure that if a business re-brands (e.g. a Shell station 
> changes to an Exxon station) someone changes all the tags and not just the 
> Name that gets rendered.

That's all well meaning but somehow I doubt that a read-only field
will achieve these goals. I see one of two things happen: a
casual mapper who just wants to fix something suddenly finds
that they cannot change stuff and just leave. And the people,
who change names just for fun will randomly poke at the user
interface until their change goes through.

I tried the latter a bit. Lets assume somebody realises that
this McDonalds is actually a Burger King:
https://www.openstreetmap.org/way/586629904

Trying to change that in the new iD, here is the things that
happened during a few minutes me poking the interface in an
attempt to achieve that goal:

* Change only Operator because it is the only editable field that contains
  the offending 'McDonalds' name.

* Change all 'McDonalds' in the raw list of tags. Leaving the wikidata
  field as is because there is no way to know that there is any
correspondence.

* Actually bother to read the tool tip that appears on the locked name,
don't understand a word it says (what is that wikidata it is speaking
of?). The only thing that parses is: 'delete it'.
Click on the closest 'dust bin' you can find(the one for the name
field). Et voila, the name field is writeable again. Conclude that
this is the proper way to change a name.

* Do the not so obvious and click on that 'Fast Food' button on top.
  List appears with preset of which none are 'Fast Food'. There is
no way to know from just looking at the interface that you have
to type 'Burger King' into the search box to achieve your goal.
Even if succeed in finding out, operator and wikipedia
remain with 'McDonalds'. Good luck noticing that.

All of these are relatively likely to happen to somebody unfamiliar
with iD and all end you in an inconsistent state despite the locked
name. The problem here is that there is nothing obvious about the
'protected' fields. As such they are no hint to the user
on how to map but they are  just a barrier to circumvent. You could
of course add more 'protections' to prevent the circumventions
but that just ends in an arms race the editor writer can't win.
All you end up with is an editor that is no longer usable as
a general purpose editor.

To add to that: data consumers (including me) like
to complain a lot about the quality of OSM data but that is
actually greatly exaggerated. As developers, we tend to mostly
see the bad data points because the 99% of good data just
gets processed quietly without drawing any attention. That
tends to skew our perception. The number of errors this
'protection change' prevents is simply not relevant in the
grand scheme of things. And people who want to do real damage
will certainly not be deterred by a tiny read-only barrier.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] pyosmium: osmium extract

2018-11-10 Thread Sarah Hoffmann
On Sat, Nov 10, 2018 at 11:24:35PM +0900, koji higuchi wrote:
> I mean if similar task could be done with  pyosmium?

There is no built-in functionality for clipping by bounding box
in pyosmium. Doing this in python would be far too slow.

If you want to process a smaller extract with pyosmium, run
'osmium extract' first and then run your pyosmium script on
the resulting file.

Sarah


> On Sat, Nov 10, 2018 at 11:23 PM Jochen Topf  wrote:
> 
> > On Sat, Nov 10, 2018 at 09:47:40PM +0900, koji higuchi wrote:
> > > There is  osmium extract command line available for osmium.
> > >
> > > osmium extract -b 11.35,48.05,11.73,48.25 germany-latest.osm.pbf \
> > > -o munich.osm.pbf
> > >
> > > How can I use it using the pyosmium?
> >
> > You don't. You use it using "osmium". That's why it is an "osmium"
> > command line. Osmium is a command line tool, PyOsmium is a Python
> > library based on the libosmium C++ library.
> >
> > Jochen
> > --
> > Jochen Topf  joc...@remote.org  https://www.jochentopf.com/
> > +49-351-31778688
> >

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] RuntimeError: need at least two points for linestring (way_id=619453148)

2018-11-04 Thread Sarah Hoffmann
Hi,

On Fri, Nov 02, 2018 at 11:36:49AM +0900, koji higuchi wrote:
> Hi Sarah,
> It stopped working with the following error, though I used try and
> exception:
> 
> Traceback (most recent call last):
>   File "D:\test.py", line 47, in 
> ShapeConverter().apply_file(fi, locations=True)
> RuntimeError: Read failed: The data is invalid.

This means that one of your input files is broken.
You need to find out which one and repair it.

Sarah

> 
> Please help me.
> 
> On Thu, Nov 1, 2018 at 7:59 PM koji higuchi  wrote:
> 
> > Hi Sarah,
> > Your answer was great and it helped me to correct my code.
> > I think that it's okay to use class and definition.
> > Thank you very much for your help.
> >
> > Best regards
> > Koji
> >
> >
> > On Thu, Nov 1, 2018 at 7:07 PM koji higuchi  wrote:
> >
> >> Hi Sarah,
> >> Thank you so much for your help!
> >> Writing speed for GPKG format with fiona has been very slow.
> >> I wanted to extract one by one ids and write geom and tag using
> >> ogr/python.
> >> Could you help me how to extract geom and tag of each id in a for loop
> >> without using class and definitions?
> >>
> >> On Thu, Nov 1, 2018 at 6:07 PM Sarah Hoffmann  wrote:
> >>
> >>> Hi,
> >>>
> >>> On Thu, Nov 01, 2018 at 12:13:40PM +0900, koji higuchi wrote:
> >>> > *I tried to extract data from .osm.pbf file and write to shapefile as
> >>> > follows:*
> >>> >
> >>> > import os, osmium, fiona
> >>> >
> >>> > fi = 'europe-latest.osm.pbf'
> >>> > fo = 'europe-latest.shp'
> >>> >
> >>> > drv = 'ESRI Shapefile'
> >>> >
> >>> > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
> >>> > 'longlat'}
> >>> >
> >>> >  schema = {'geometry': 'LineString',
> >>> >'properties': {'id': 'float', 'name' : 'str',
> >>> 'kind' :
> >>> > 'str'}}
> >>> >
> >>> > outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
> >>> >
> >>> > geomfab = osmium.geom.GeoJSONFactory()
> >>> >
> >>> > class ShapeConverter(osmium.SimpleHandler):
> >>> >   def way(self, w):
> >>> >if 'place' in w.tags:
> >>> >rec = {'geometry' : eval(geomfab.create_linestring(w)),
> >>> >   'properties' : {'id' : float(w.id),
> >>> >  'name' : w.tags.get('name'),
> >>> >'kind' : w.tags['place']}}
> >>> >outfile.write(rec)
> >>> >
> >>> > ShapeConverter().apply_file(fi, locations=True)
> >>> >
> >>> > I got the following error after extracting several contents:
> >>> >
> >>> >  rec = {'geometry' : eval(geomfab.create_linestring(w)),
> >>> > RuntimeError: need at least two points for linestring
> >>> (way_id=619453148)
> >>> >
> >>> > How could I skip that erroneous id and extract data for other working
> >>> ids?
> >>>
> >>> You need to do this manually yourself in the handler. I recommend
> >>> simply catching the exception as there are some other error
> >>> conditions besides too few points:
> >>>
> >>> def way(self, w):
> >>> if 'place' in w.tags:
> >>> try:
> >>> geom = geomfab.create_linestring(w)
> >>> except:
> >>> print("Skipping way with bad geometry")
> >>> return
> >>>
> >>>   rec = {'geometry' : eval(geom),
> >>>'properties' : {'id' : float(w.id),
> >>>'name' : w.tags.get('name'),
> >>>'kind' : w.tags['place']}}
> >>> outfile.write(rec)
> >>>
> >>> Kind regards
> >>>
> >>> Sarah
> >>>
> >>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] RuntimeError: need at least two points for linestring (way_id=619453148)

2018-11-01 Thread Sarah Hoffmann
Hi,

On Thu, Nov 01, 2018 at 12:13:40PM +0900, koji higuchi wrote:
> *I tried to extract data from .osm.pbf file and write to shapefile as
> follows:*
> 
> import os, osmium, fiona
> 
> fi = 'europe-latest.osm.pbf'
> fo = 'europe-latest.shp'
> 
> drv = 'ESRI Shapefile'
> 
> crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
> 'longlat'}
> 
>  schema = {'geometry': 'LineString',
>'properties': {'id': 'float', 'name' : 'str', 'kind' :
> 'str'}}
> 
> outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
> 
> geomfab = osmium.geom.GeoJSONFactory()
> 
> class ShapeConverter(osmium.SimpleHandler):
>   def way(self, w):
>if 'place' in w.tags:
>rec = {'geometry' : eval(geomfab.create_linestring(w)),
>   'properties' : {'id' : float(w.id),
>  'name' : w.tags.get('name'),
>'kind' : w.tags['place']}}
>outfile.write(rec)
> 
> ShapeConverter().apply_file(fi, locations=True)
> 
> I got the following error after extracting several contents:
> 
>  rec = {'geometry' : eval(geomfab.create_linestring(w)),
> RuntimeError: need at least two points for linestring (way_id=619453148)
> 
> How could I skip that erroneous id and extract data for other working ids?

You need to do this manually yourself in the handler. I recommend
simply catching the exception as there are some other error
conditions besides too few points:

def way(self, w):
if 'place' in w.tags:
try:
geom = geomfab.create_linestring(w)
except:
print("Skipping way with bad geometry")
return

  rec = {'geometry' : eval(geom),
   'properties' : {'id' : float(w.id),
   'name' : w.tags.get('name'),
   'kind' : w.tags['place']}}
outfile.write(rec)

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] osm2pgsql release 0.96.0

2018-05-02 Thread Sarah Hoffmann
Hi,

we are happy to announce a new release 0.96.0 of osm2pgsql.

This release mostly fixes a number of regressions introduced
with the switch to libosmium and brings a couple of improvements
in the build system. Contrary to what was announced for the
last version, this release still supports old-style multipolygons.

Changes include

- memory for caches and flatnode storage is freed earlier, leaving
  more RAM to Postgresql during indexing

- extend web Mercator to 89.99 latitude again, reducing broken polygons

- skip objects with no tags during initial import, improving
  performance during first import stage

- support LuaJIT for faster processing of Lua tag transforms
  (thanks to @mmd)

- update to libosmium 2.14

- windows builds for 32bit are now provided via Appveyor
  (thanks to @alirdn)

- bug fixes for tile expiry (thanks to @nakaner)

For anybody building against an external libosmium, please note
that you now also need to configure an external protozero library
separately.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] [OSM-talk] Nominatim Installation Issues on Ubuntu 17.10(GSOC)

2018-03-05 Thread Sarah Hoffmann
Hi Rahul,

welcome to OSM GSoC and thank you for your interest in the Nominatim
projects. I'm glad to hear that you managed to get the development
environment set up. We generally recommend to use the Vagrant scripts
for a first try, which is still based on Ubuntu 16.04LTS but there
should not be any significant difference with Ubuntu 17.10.

With the development environment set up, I recommend that you next
import an extract from your local area and get used to the API. Then
you should take some time to get used to the code. Have a look at the
issues marked simple[1] and see if you can solve them.

If you have any questions, feel free to ask here on the dev@ mailing
list or on the more specific geocoding@ mailing list.

Kind regards

Sarah
(lonvia)

[1] https://github.com/openstreetmap/Nominatim/labels/simple

On Tue, Mar 06, 2018 at 12:19:29AM +0530, Rahul Soshte wrote:
> The reason I wanna set up Nominatim because I am gonna choose one of the
> ideas listed here
> 
> in the Nominatim sub-section for GSOC.
> 
> On Tue, Mar 6, 2018 at 12:11 AM, Rahul Soshte 
> wrote:
> 
> > Hi,
> > This is Rahul Soshte,a 3rd Year Computer Engineering Student from Mumbai
> > University(India).I am trying to setup the development environment of
> > Nominatim on my Ubuntu 17.10 and I am facing some issues.I am willing to
> > contribute to the organization through GSOC(whose completion may not deter
> > my conributions to the org) by contributing to Nominatim.So to get started
> > I want to setup the development environment for Nominatim but I am facing
> > some issues like the ones seen in the screenshots attached.What do I do?
> >
> >
> >
> >

> ___
> talk mailing list
> t...@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim 3.1.0 released

2018-01-17 Thread Sarah Hoffmann
Hi,
  
we are happy to announce the release of Nominatim 3.1.0,
the OpenStreetMap geocoding software.

This release brings a major overhaul of postcode handling and the
PHP frontend code.

Postcodes now reside in their own table `location_postcode` which can be
updated at any time. Postcodes for places are no longer part of the
standard address but are guessed with using a dedicated algorithm. On the
search side, post codes can now be ignored in the query, improving results
in areas where postcode coverage in OSM is bad. For more details read 
https://www.openstreetmap.org/user/lonvia/diary/43143.

The PHP frontend code that parses incoming queries and looks up and
formats the results has been thoroughly refactored into a set of classes
with much smaller functions. All code is now documented and the API tests
have resulted in almost 100% code coverage. 

For a more complete list of changes, see the ChangeLog file.

For users of Nominatim 3.0.0, there is a migration guide at
http://nominatim.org/release-docs/latest/admin/Migration/ However,
to fully benefit from the improved postcodes, a database reimport is
strongly recommended.

Installation instructions can be found at
http://nominatim.org/release-docs/latest/

Happy geocoding.

Sarah


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [osmosis-dev] Osmosis update failed: Pipeline entities are not sorted or contain multiple versions of a single entity

2017-12-28 Thread Sarah Hoffmann
Hi,

On Wed, Dec 27, 2017 at 08:42:45PM -0500, do...@mail.com wrote:
> Hello,
> 
> I'm getting an out of order error from osmosis and I'm not
> certain what to do about it.
> I got the update from an offical mirror, so osmosis should
> not have any problems with it.
> I am using the latest release of osmosis as far as I know.
> I have looked at the xml change file and there are 2 entities with an
> identical value for the "id" attribute, but different values "version"
> attribute.

You need to filter those multiple versions before applying a
change to an osm file. Try --simplify-change on the change file[1].

Kind regards

Sarah

[1] 
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.46#--simplify-change_.28--simc.29

> 
> Thanks!
> 
> 
> % nice -n15 lbzip2 -kdc planet-160718.osm.bz2 | nice
> -n5 /home/me/Documents/programs/osmosis-bin/bin/osmosis --read-xml-change
> file=406.osc.gz --buffer-change bufferCapacity=12000 --fast-read-xml
> file=/dev/stdin compressionMethod=none --buffer bufferCapacity=12000
> --apply-change --buffer bufferCapacity=12000 --write-xml file=/dev/stdout
> | pv -arb | nice -n15 lbzip2 -c9z > planet.osm.bz2 Dec 26, 2017 10:31:47
> AM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.45
> Dec 26, 2017 10:31:47 AM org.openstreetmap.osmosis.core.Osmosis run INFO:
> Preparing pipeline. Dec 26, 2017 10:31:47 AM
> org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline
> execution. Dec 26, 2017 10:31:48 AM
> org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing,
> waiting for completion. Dec 26, 2017 10:35:06 AM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion SEVERE: Thread for task 1-read-xml-change failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error
> has occurred, aborting. at
> org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:162)
> at
> org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:218)
> at
> org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:305)
> at
> org.openstreetmap.osmosis.core.buffer.v0_6.ChangeBuffer.process(ChangeBuffer.java:48)
> at
> org.openstreetmap.osmosis.xml.v0_6.impl.ChangeSourceElementProcessor$ChangeSinkAdapter.process(ChangeSourceElementProcessor.java:144)
> at
> org.openstreetmap.osmosis.xml.v0_6.impl.NodeElementProcessor.end(NodeElementProcessor.java:139)
> at
> org.openstreetmap.osmosis.xml.v0_6.impl.OsmChangeHandler.endElement(OsmChangeHandler.java:94)
> at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
> at
> org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown
> Source) at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source) at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source) at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at
> org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at
> org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at
> javax.xml.parsers.SAXParser.parse(SAXParser.java:195) at
> org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:90)
> at java.lang.Thread.run(Thread.java:748)
> 
> Dec 26, 2017 10:35:06 AM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion SEVERE: Thread for task 2-buffer-change failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities
> are not sorted or contain multiple versions of a single entity, previous
> entity type=Node, id=7040756, version=5 current entity type=Node,
> id=7040756, version=6. at
> org.openstreetmap.osmosis.core.sort.v0_6.SortedDeltaChangePipeValidator.process(SortedDeltaChangePipeValidator.java:58)
> at
> org.openstreetmap.osmosis.core.buffer.v0_6.ChangeBuffer.run(ChangeBuffer.java:84)
> at java.lang.Thread.run(Thread.java:748)
> 
> Dec 26, 2017 10:35:06 AM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion SEVERE: Thread for task 3-fast-read-xml failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read
> XML file /dev/stdin. at
> org.openstreetmap.osmosis.xml.v0_6.FastXmlReader.run(FastXmlReader.java:102)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> org.openstreetmap.osmosis.core.OsmosisRuntimeException:
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error
> has occurred, aborting. at
> org.openstreetmap.osmosis.xml.v0_6.impl.FastXmlParser.readOsm(FastXmlParser.java:436)
> at
> 

Re: [OSM-dev] Pyosmium relations

2017-11-24 Thread Sarah Hoffmann
Hi,

On Fri, Nov 24, 2017 at 07:14:00PM +, Xavier Barnada wrote:
> Hi,
> 
> I am working on an aplication to detect specific changes in an area (
> https://github.com/Xevib/changewithin ).
> 
> My doubt is, is there any way to know the lat and lon of each node of a
> relation usign pyosmium?

Pyosmium can only process multipolygon relations into geometries
at the moment (via the area() callback). This might be enough in
your case, if you are really only interested in buildings. Have
a look at the amenity_list.py example to see how to get the
geometry as a Shaply object.

There is no generic mechanism for handling geometries, mainly because
it is rather difficult to come up with a truely generic approach.
Relations can represent a lot of diferent concepts and each needs
a slightly different handling. In your case, for example, I suspect
that you not only want lat/lon of the nodes directly in the relation
but of the nodes of the ways in the relations.

> 
> I tried to solve this problem making "cache" of the nodes on the file but
> this does not seem to be a good strategy

That's essentially what happens when pyosmium processes multipolygon
relations into areas. A node cahce is always needed when working with
OSM data.

I see that you plan to process change files. Please keep in mind
that these files do not contain the complete node or way information
reuqired to obtain the geometry for relations. If you require geometry
information of ways or relations, you need to start from the planet
file (or an excerpt of the area you are interested in) and keep the
current planet state to look up missing information.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Read osmChange files with Pyosmium

2017-03-18 Thread Sarah Hoffmann
Hi again,

looking at the code again, it turns out changesets are actually
implemented.

I suspect that you are mixing up terminology here. A changeset
in osmium, is the collection of metadata about a particular
set of data that was uploaded. Changeset information is
available in a separate file here:
http://planet.osm.org/planet/changesets-latest.osm.bz2

The XML for changesets looks something like that:






You probably want to process OSM data change files from
http://planet.osm.org/replication/. These files are normal
data files. so you have to process them using the data
callbacks node(), way(), relation(). Have a look at the
osm_diff_stats.py example, to find out how to distinguish
between created, modified and deleted objects.

Kind regards

Sarah

On Sat, Mar 18, 2017 at 08:20:26AM +, Sarah Hoffmann wrote:
> Hi Giovanni,
> 
> the ChangeSet type is currently not exported into a python type
> but it's really easy to do. Could you open a feature request at
> https://github.com/osmcode/pyosmium/issues and I'll see to it to
> add support for it.
> 
> Kind regards
> 
> Sarah
> 
> 
> On Fri, Mar 17, 2017 at 12:33:01PM +0100, G. Allegri wrote:
> > I've just started giving a look to Pyosmium. I've done tests based on the
> > examples from its repo and everything works fine.
> > I would like to replicate osmium changeset-filter [1] and I'm striving to
> > make the osmium.SimpleHandler parse the changesets from an osmChange file.
> > 
> > I've tried:
> > 
> > >>>class MyHandler(osmium.SimpleHandler):
> > ... def changeset(self,changeset):
> > ... print changeset
> > 
> > >>>handler = MyHandler()
> > >>>handler.apply_file('mychange.xml>')
> > 
> > but nothing happens. No errors, no outputs.
> > 
> > I've tried to setup the osmium.io.Reader manually, and setting the entity
> > bit:
> > 
> > >>>reader =
> > osmium.io.Reader('mychange.xml',osmium.osm.osm_entity_bits.CHANGESET)
> > >>>omsium.apply(reader,handler)
> > 
> > No output again.
> > 
> > First question: is it possible to parse osmChanges with Pyosmium? If yes,
> > what's the right way to do it?
> > 
> > Thanks a lot,
> > Giovanni
> > 
> > 
> > [1] http://osmcode.org/osmium-tool/manual.html#working-with-changesets
> 
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
> 

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Read osmChange files with Pyosmium

2017-03-18 Thread Sarah Hoffmann
Hi Giovanni,

the ChangeSet type is currently not exported into a python type
but it's really easy to do. Could you open a feature request at
https://github.com/osmcode/pyosmium/issues and I'll see to it to
add support for it.

Kind regards

Sarah


On Fri, Mar 17, 2017 at 12:33:01PM +0100, G. Allegri wrote:
> I've just started giving a look to Pyosmium. I've done tests based on the
> examples from its repo and everything works fine.
> I would like to replicate osmium changeset-filter [1] and I'm striving to
> make the osmium.SimpleHandler parse the changesets from an osmChange file.
> 
> I've tried:
> 
> >>>class MyHandler(osmium.SimpleHandler):
> ... def changeset(self,changeset):
> ... print changeset
> 
> >>>handler = MyHandler()
> >>>handler.apply_file('mychange.xml>')
> 
> but nothing happens. No errors, no outputs.
> 
> I've tried to setup the osmium.io.Reader manually, and setting the entity
> bit:
> 
> >>>reader =
> osmium.io.Reader('mychange.xml',osmium.osm.osm_entity_bits.CHANGESET)
> >>>omsium.apply(reader,handler)
> 
> No output again.
> 
> First question: is it possible to parse osmChanges with Pyosmium? If yes,
> what's the right way to do it?
> 
> Thanks a lot,
> Giovanni
> 
> 
> [1] http://osmcode.org/osmium-tool/manual.html#working-with-changesets

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Requesting for further instructions

2017-03-17 Thread Sarah Hoffmann
Hi Subhani,

thanks for your interest in GSoC. I can give some hints about the
Nominatim projects.

As a first step, you should try to set up a database with OSM data
of a region you are familiar with. http://download.geofabrik.de/
provides reasonably sized extracts you can use. Then play around
with the interface and try to get an understanding of the relation
between the OSM data and the data presented by Nominatim.

For the Reverse geocoding project, analyse the current reverse
geocoding algorithm (lib/ReverseGeodocde.php) and then go through
the issues in github and trac to get an overview, what are the current
shortcomings. Adding a few tests in test/bdd/api/reverse/ is also
a great way to get into the code.

For the postcode project, you should look mainly into the import
process, find out how postcodes are currently handled (starting
point would be the calculate-postcodes section in utils/import.php).
Also you need to get familiar with postcode tagging in OSM and
do some reasearch what different postcode systems are in use al
over the world.

If you want to get your hands dirty with code in general, there
are some open PRs that need to be finished, in particular #552,
#622 and #624 are good for getting started.

Kind regards

Sarah

On Sat, Mar 18, 2017 at 12:31:44AM +0530, Subhani Munasinghe wrote:
> Hi everyone,
> 
>  As I was given some instructions to select a project idea, I went
> through the project idea list in
> 
> https://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2017/Project_Ideas
> .
> 
>  I am very interested in three project ideas there. Even though I have
> to select one idea eventually, I would love to know about all those three.
> These are the three projects in I am interested and I would love to
> contribute to one of them.
> 
>1. Improve Postcode Handling (under Nominatim)
>2. Improve Reverse Geocoding Algorithm for Nominatim (under Nominatim)
>3. Make the website use the API (under API)
> 
>   I have done some university project works using *PHP* and I am
> currently self learning *postgresql*. Also, I have completed my 5 months
> internship and there I worked in a project in which *Java, SQL *and* Git* were
> heavily used.
> 
>  What I have done up to now is, created an account in www.osm.org  and
> mapped some data. I got a very good basic idea about how the map grows.
> Also I read the documentation about *imports* and got a brief idea about
> how it works. I practiced how to add objects into map like points, roads,
> areas, tags etc. And it is very very interesting.
> 
>   Now, what I want to know is how to start to do some work in order to
> get familiar with the above mentioned 3 projects. I already forked and
> cloned the source code repository of *Nominatim *from
> https://github.com/openstreetmap/Nominatim. But I didn't try any coding
> because I don't have any idea. So what should be my next step ?
> 
> Thanks in advance !

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] GSoC 17 : Nominatim- Add Wikidata to Nominatim

2017-03-15 Thread Sarah Hoffmann
Hi Amisha,

thanks for your intersted in GSoC.

The Nominatim source code is a good start. You should try to set up your
own database using a small excerpt of a region you are familiar with
(ready made excerpts of OSM data are available at
http://download.geofabrik.de/). Play a bit around with the database
and try to understand how OSM objects and their tags are used in
the database schema.

For the wikidata project you should pay particular attention to
the import process. Have a look at the preprocessed wiki data we
provide (http://www.nominatim.org/data/wikipedia_article.sql.bin
and http://www.nominatim.org/data/wikipedia_redirect.sql.bin).
Recreating updated versions of these files will be the first task
of the GSoC project.

I would als recommend that you have a look at the Wikipedia and
Wikidata dumps and research a bit what tools for processing are
available.

I hope that helps you getting started. If you have questions,
I recommend that you subscribe to the geocoding mailinglist
(https://lists.openstreetmap.org/listinfo/geocoding)

Kind regards

Sarah


On Mon, Mar 13, 2017 at 07:30:29PM +0530, amisha budhiraja wrote:
> Hi all,
> I am Amisha Budhiraja, third-year Computer Science Engineering student at
> Guru Nanak Dev Engineering College, Punjab.
> I wish to participate in Google Summer of Code this year. I was interested
> in OSM, since I was doing my six-weeks training in it. I have made complete
> non-interactive OSM installation script
> ​ and many other tasks​
> .
> I downloaded the Nominatim source code from the github repository [1].
> ​ ​
> I am skilled with postgresql and php and would love to contribute on the
> Nominatim.
> ​ ​
> I love to improve the existing importance ranking​.​
> Is there any other resources that I should follow in the meantime, that
> would be useful for this project?
> 
> [1]. https://github.com/openstreetmap/Nominatim
> 
> -- 
> Regards,
> Amisha Budhiraja.
> Blog: https://amisha2016.wordpress.com
> Github: https://github.com/amisha2016
> Gitlab: https://gitlab.com/u/Amisha2016
> The human face has limited space. If you fill it with laughter there will
> be no room for crying.

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Submitting proposal for GSoC 2017

2017-02-21 Thread Sarah Hoffmann
Hi,

a few thinkgs you can check:

- has the Apache configuration in /etc/apache2/conf-available/nominatim.conf
  been created correctly and does it point to the correct directory
  (website in your build directory)?
- try restarting apache
- check that user 'www-data' can access the website directory

Kind regards

Sarah

On Tue, Feb 21, 2017 at 05:22:34PM +0530, Hriday N Sanghvi wrote:
> Hello Sarah,
> 
> 
> Thank you for recommending a solution.
> 
> As suggested, I have successfully installed:
> 
> 1.   Vagrant (https://releases.hashicorp.com/vagrant/1.9.1/vagrant_1.9.
> 1.msi)
> 
> 2.   Oracle Virtual Box (http://download.virtualbox.
> org/virtualbox/5.1.14/VirtualBox-5.1.14-112924-Win.exe).
> 
> 3.   Official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build (
> https://atlas.hashicorp.com/ubuntu)
> 
> 4.   Nominatim with dependencies using https://github.com/twain47/
> Nominatim/blob/master/docs/install-on-ubuntu-16.md
> 
> 5.   Monaco data extract (http://download.geofabrik.de/
> europe/monaco-latest.osm.pbf) using https://github.com/twain47/
> Nominatim/blob/master/docs/Import_and_update.md and
> https://github.com/twain47/Nominatim/blob/master/VAGRANT.md
> 
> [image: Inline image 1]
> 
> On accessing localhost:8089, I get the following page:
> 
> [image: Inline image 2]
> 
> However, on accessing localhost:8089/nominatim, I get an error as shown
> below:
> 
> [image: Inline image 3]
> 
> I tried solving the issue by referring to https://github.com/twain47/
> Nominatim/issues/203, but the problem still persists. I would appreciate if
> you could suggest possible solutions. In the mean time, I’m browsing
> through other similar issues on the Nominatim Github repository pages for
> workarounds.
> 
> 
> Thank you.
> 
> 
> Regards,
> 
> Hriday
> 
> On Sat, Feb 18, 2017 at 5:28 PM, Sarah Hoffmann <lon...@denofr.de> wrote:
> 
> > Hi Hriday,
> >
> > the minimum memory requirement was 2GB RAM last time I checked. You won't
> > need more than that when working with smaller extracts. If anyhow
> > possible I recommend to try the installation on your local machine.
> > There are Vagrant script included, if you prefer to run it on a
> > virtual machine.
> >
> > Kind regards
> >
> > Sarah
> >
> >
> > On Sat, Feb 18, 2017 at 10:56:32AM +0530, Hriday N Sanghvi wrote:
> > > Hello Sarah,
> > >
> > >
> > > Thank you for guiding me through the first steps in developing an
> > > implementation plan for the project titled “Improve Postcode Handling”.
> > >
> > >
> > > I have gone through http://wiki.openstreetmap.org/wiki/Elements to
> > better
> > > understand the OSM data model. I now have a fair idea of what nodes,
> > ways,
> > > relations, tags are and how they are related to each other. I also went
> > > through http://wiki.openstreetmap.org/wiki/Key:addr and
> > > http://wiki.openstreetmap.org/wiki/Tag:boundary%3Dpostal_code and
> > > understand how postcodes are either mapped to an address or to an area
> > > within a geo fence. I also built and ran a few addr:postcode and
> > >  boundary:postal_code queries using the Overpass Turbo Wizard (
> > > https://overpass-turbo.eu/ )for examples of the data output on the map.
> > >
> > >
> > > I went through https://en.wikipedia.org/wiki/Postal_code to get a
> > general
> > > idea about postcode systems. The characters used in postal codes are the
> > > Arabic numerals "0" to "9", letters of the ISO basic Latin alphabet,
> > spaces
> > > and hyphens. India has a 6 character numeric postcode, but many countries
> > > have alphanumeric postcodes with varying number of characters.
> > >
> > >
> > > I have experience in installing Wordpress, WooCommerce and other
> > Ecommerce
> > > softwares on AWS. Hence, I plan to install Nominatim on AWS Server - free
> > > tier (https://aws.amazon.com/free/ ). However, from what I’ve read, it
> > > looks like 1 GB of RAM provided under free tier may not be sufficient. I
> > > would appreciate if you could suggest alternative low cost/free servers.
> > In
> > > addition to
> > > https://github.com/twain47/Nominatim/blob/master/docs/
> > install-on-ubuntu-16.md
> > > and
> > > https://github.com/twain47/Nominatim/blob/master/docs/
> > Import_and_update.md
> > > , I will also refer to the following installation guides:
> > > https://www.snip2code.com/Snippet/248983/Instructions-
> &g

Re: [OSM-dev] Submitting proposal for GSoC 2017

2017-02-18 Thread Sarah Hoffmann
Hi Hriday,

the minimum memory requirement was 2GB RAM last time I checked. You won't
need more than that when working with smaller extracts. If anyhow
possible I recommend to try the installation on your local machine.
There are Vagrant script included, if you prefer to run it on a
virtual machine.

Kind regards

Sarah


On Sat, Feb 18, 2017 at 10:56:32AM +0530, Hriday N Sanghvi wrote:
> Hello Sarah,
> 
> 
> Thank you for guiding me through the first steps in developing an
> implementation plan for the project titled “Improve Postcode Handling”.
> 
> 
> I have gone through http://wiki.openstreetmap.org/wiki/Elements to better
> understand the OSM data model. I now have a fair idea of what nodes, ways,
> relations, tags are and how they are related to each other. I also went
> through http://wiki.openstreetmap.org/wiki/Key:addr and
> http://wiki.openstreetmap.org/wiki/Tag:boundary%3Dpostal_code and
> understand how postcodes are either mapped to an address or to an area
> within a geo fence. I also built and ran a few addr:postcode and
>  boundary:postal_code queries using the Overpass Turbo Wizard (
> https://overpass-turbo.eu/ )for examples of the data output on the map.
> 
> 
> I went through https://en.wikipedia.org/wiki/Postal_code to get a general
> idea about postcode systems. The characters used in postal codes are the
> Arabic numerals "0" to "9", letters of the ISO basic Latin alphabet, spaces
> and hyphens. India has a 6 character numeric postcode, but many countries
> have alphanumeric postcodes with varying number of characters.
> 
> 
> I have experience in installing Wordpress, WooCommerce and other Ecommerce
> softwares on AWS. Hence, I plan to install Nominatim on AWS Server - free
> tier (https://aws.amazon.com/free/ ). However, from what I’ve read, it
> looks like 1 GB of RAM provided under free tier may not be sufficient. I
> would appreciate if you could suggest alternative low cost/free servers. In
> addition to
> https://github.com/twain47/Nominatim/blob/master/docs/install-on-ubuntu-16.md
> and
> https://github.com/twain47/Nominatim/blob/master/docs/Import_and_update.md
> , I will also refer to the following installation guides:
> https://www.snip2code.com/Snippet/248983/Instructions-for-installing-an-OpenStree
> , https://andrewwhitby.com/2014/12/18/nominatim-on-ec2/ ,
> http://wiki.openstreetmap.org/wiki/Nominatim/Installation .
> 
> Thank you.
> 
> Regards,
> 
> Hriday.
> 
> On Fri, Feb 17, 2017 at 2:23 AM, Sarah Hoffmann <lon...@denofr.de> wrote:
> 
> > Hi Hriday,
> >
> > thank you for your interest in the OSM GSoC.
> >
> > There are a couple of things you can do to prepare for the postcode
> > project.
> >
> > First of all you should familiarize yourself a bit with the OSM data
> > model in general (nodes, ways, relations, tags) and then find out
> > how postcode tagging works in particular. There are essetially two
> > ways how postcodes are mapped: as part of an address for a particular
> > house (see http://wiki.openstreetmap.org/wiki/Key:addr) or as an
> > area (see http://wiki.openstreetmap.org/wiki/Tag:boundary%3Dpostal_code).
> > You can use Overpass (or Overpass Turbo) to find examples of the data.
> >
> > Nominatim actually uses complete OSM data dumps and processes them
> > into a search database. So improving the search means improving the
> > import process. So, next you should try and set up your own instance
> > of Nominatim using the current development version from
> > https://github.com/twain47/Nominatim (installation documentation
> > can be found in the doc/ directory) and a data extract from
> > http://download.geofabrik.de/. I recommend something from Europe
> > as there is already a fair coverage with postcodes.
> >
> > Finally, it will also be helpful if you do a little bit of research
> > about postcodes in general to find out what different systems are
> > in use and where.
> >
> > Hope that helps to get you started.
> >
> > Kind regards
> >
> > Sarah
> >
> > On Thu, Feb 16, 2017 at 07:13:31PM +0530, Hriday N Sanghvi wrote:
> > > Hello Lonvia,
> > >
> > >
> > > I’m a student of Computer Science Engineering at SRM University, India (
> > > http://www.srmuniv.ac.in/). I am interested in submitting my proposal
> > for
> > > GSOC 2017 for either one of the Open Street Map projects 1. Improve
> > > Postcode handling or 2. OpenAddresses for Nominatim.
> > >
> > >
> > > I have extensive experience working with PHP and MySQL databases on
> > various
> > > web based projects. I 

Re: [OSM-dev] Submitting proposal for GSoC 2017

2017-02-16 Thread Sarah Hoffmann
Hi Hriday,

thank you for your interest in the OSM GSoC.

There are a couple of things you can do to prepare for the postcode
project.

First of all you should familiarize yourself a bit with the OSM data
model in general (nodes, ways, relations, tags) and then find out
how postcode tagging works in particular. There are essetially two
ways how postcodes are mapped: as part of an address for a particular
house (see http://wiki.openstreetmap.org/wiki/Key:addr) or as an
area (see http://wiki.openstreetmap.org/wiki/Tag:boundary%3Dpostal_code).
You can use Overpass (or Overpass Turbo) to find examples of the data.

Nominatim actually uses complete OSM data dumps and processes them
into a search database. So improving the search means improving the
import process. So, next you should try and set up your own instance
of Nominatim using the current development version from
https://github.com/twain47/Nominatim (installation documentation
can be found in the doc/ directory) and a data extract from
http://download.geofabrik.de/. I recommend something from Europe
as there is already a fair coverage with postcodes.

Finally, it will also be helpful if you do a little bit of research
about postcodes in general to find out what different systems are
in use and where.

Hope that helps to get you started.

Kind regards

Sarah

On Thu, Feb 16, 2017 at 07:13:31PM +0530, Hriday N Sanghvi wrote:
> Hello Lonvia,
> 
> 
> I’m a student of Computer Science Engineering at SRM University, India (
> http://www.srmuniv.ac.in/). I am interested in submitting my proposal for
> GSOC 2017 for either one of the Open Street Map projects 1. Improve
> Postcode handling or 2. OpenAddresses for Nominatim.
> 
> 
> I have extensive experience working with PHP and MySQL databases on various
> web based projects. I have undergone a PostgreSQL course on
> https://www.lynda.com/PHP-5-tutorials/PostgreSQL-9-with-PHP-Essential-Training/73930-2.html
> and http://www.tutorialspoint.com/postgresql/ and I’m quite comfortable
> working with Github. I have some experience with Google Maps API while
> working on a location based Click-and-Collect grocery shopping app. I plan
> to continue with coding for OSM even after the GSOC project and also plan
> to base my six months project in the eighth and final semester of my course
> on OSM.
> 
> I understand that the list of accepted mentoring organizations will be
> published on February 27 and potential student participants will discuss
> application ideas with mentoring organizations from February 27 to March 20
> and that student application period is open from March 20 to April 3, but I
> wish to start early.
> 
> 
> I have downloaded JOSM to edit my local area on the map, but I found ID
> editor to be an easier option.
> 
> With regard to extraction of postcode data and exporting data to the
> Nominatim database related to the project titled “Improve Postcode
> Handling”, I have looked at API v 0.6 (
> http://wiki.openstreetmap.org/wiki/API_v0.6 ), including the endpoints for
> capabilities (http://api06.dev.openstreetmap.org/api/capabilities ),
> permissions (http://api06.dev.openstreetmap.org/api/0.6/permissions ),
> change sets (http://api06.dev.openstreetmap.org/api/0.6/changesets ), nodes
> (http://api06.dev.openstreetmap.org/api/0.6/node/12345 ), and trace
> metadata (http://api06.dev.openstreetmap.org/api/0.6/gpx/12345/details ) in
> which I was asked for Authentication.
> 
> I also went through the Overpass API (
> http://wiki.openstreetmap.org/wiki/Overpass_API ), Overpass Turbo remote (
> http://overpass-turbo.eu/ ) and Overpass Turbo Query form (
> http://www.overpass-api.de/query_form.html ).
> 
> Though I was able to download limited data using the “Download Map data
> from OSM server” tool in JOSM, I am not sure about the endpoint and query
> to get post codes using API v 0.6 or Overpass API. I would really
> appreciate any help on this. 
> 
> I’m confident of producing productive work during the GSOC period. Before I
> start working on the proposal for GSOC 2017, I would appreciate any
> guidance you could provide.
> 
> 
> Thank you.
> 
> 
> Regards,
> 
> Hriday N Sanghvi

> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] [OSM-talk] Make Nominatim more dev friendly

2017-02-01 Thread Sarah Hoffmann
Hi Mariusz,

(cross-posting to talk removed, as this is essentially a dev mail)

I'm glad to hear that you are concerned about Nominatim development.
That makes two of us. As a software developer, the most effective
way to change things is to start contributing code. So here are a few
pointers for that.

The outstanding pull requests you mention are a very good place to start.
There are quite a few which have not been merged because there are
outstanding comments from me which the original authors never addressed.
In particular, I'd like to point to:

https://github.com/twain47/Nominatim/pull/552
https://github.com/twain47/Nominatim/pull/439
https://github.com/twain47/Nominatim/pull/429

They are pretty far along. They would need to be updated to the current
master version and have the remaining issues fixed.

If that's not to your liking you can also look through the issues.
Anything marked 'enhancement' is particularly suited for external
contributions. I haven't marked the difficulty level but here are
a few examples, I'd consider good starting points for first time
code contributors:

https://github.com/twain47/Nominatim/issues/562
https://github.com/twain47/Nominatim/issues/135
https://github.com/twain47/Nominatim/issues/171
https://github.com/twain47/Nominatim/issues/255
https://github.com/twain47/Nominatim/issues/344
https://github.com/twain47/Nominatim/issues/311

The comments on the issues can be sparse at times, so feel free to ask
for clarifications. As a general rule, it is also a good idea to quickly
outline your implementation idea first, in particular where the solution
is not obvious or where larger changes are required. That helps avoid
disappointment during PR review.

If you have general questions about the source code, the geocoding@
mailing list is the right place to ask.

Sarah
(Nominatim lead developer)


On Wed, Feb 01, 2017 at 07:34:53PM +0100, Mariusz Rogowski wrote:
> blockquote {padding-left: 1ex; margin: 0px 0px 0px 0.8ex; border-left: 
> #cc 1px solid;} p {margin: 0px;padding: 0px;} 
> Hi,I am not sure if this is right pleace to rise my concerns or if 
> they are welcomed here. But I will give it a try.I am not an 
> active member of community, I am software developer who sporadically has to 
> geocode some addresses. For some regions of the world OSM is the best source 
> of data and it is a shame tools for searching the data fall behind. In my 
> opinion Nominatim could have been very useful service which would promote OSM 
> usage. Seriously, for many applications 
> searching addresses is very important feature. Nominatim should be like 
> second most important service given to the world by OSM. Unfortunately it 
> seems to be far away from the spotlight and people might not be aware of its 
> problems. What I mean is:1. There are pull requests (i.e. probably 
> finished features ready to integrate with project) starting from year 2012. 
> Yes - somebody contributed to the project and is wating 5 years to have his 
> contribution accepted. [1]2. There are over 
> 100 issues opened starting from 2013. [2]3. Project is understaffed 
> (which I guess can happen). But its maintainers are aware of it and do not do 
> anything to change it. [3]Anyway, I work in software development 
> and I could be contributing to the project. But fact the contributions are 
> ignored, maintainers are frustrated (and it shows) make me thing it is a 
> waste of time. Even providing real life examples of wrong geocoding (so the 
> test cases could be extended) ends with some 
> unfriendliness and ignorance. [4] I 
> understand there are probably valid reasons for current state and atitude. 
> But discussing it does not really interests me. I wish to change things ;) 
> Unfortunately there's not much I can do about it apart from pointing the 
> problems to wider audience. Anyway, I think the solutions to the 
> problems are quite obvious. How can I convince someone to make the project 
> open and friendly to new collaborators? />Mariusz[1] - https://github.com/twain47/Nominatim/pulls />[2] - https://github.com/twain47/Nominatim/issues[3] - 
> https://github.com/twain47/Nominatim/issues/316#issuecomment-147111016 />[4] - https://github.com/twain47/Nominatim/issues/467
> 
> 
> 

> ___
> talk mailing list
> t...@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim and/or a fault-tolerant geocoder

2016-11-29 Thread Sarah Hoffmann
Hi,

On Tue, Nov 29, 2016 at 12:03:35AM +0100, Tom wrote:
> I’m in the quest for a geocoder for OSM that is fault-tolerant in regards of 
> miss-spelled search terms.
> 
> The company I’m working for does different projects for customers in the 
> logistics field. From every customer we receive several hundred thousand 
> address-records, which we have to geocode in order to do different 
> calculations. I started to use Nominatim for that (on an own installation), 
> but it seems that Nominatim has not much of tolerance regarding miss-spelled 
> street and city names. Especially on our last project in Russia it turned 
> out, that street- and city-names often include abbreviations in different 
> ways (like „street“, „str.“, „s“, …). Since we receive the address 
> information from our customers, we have not much influence on the quality of 
> the data. So there are not just these valid abbreviations, but also real 
> spelling errors. Nevertheless we have to geocode as much of these addresses 
> as possible. 
> 
> But right now, Nominatim throws out around 40% of the addresses, not finding 
> anything, although the address is in OSM and could be found (just slightly 
> different spelled). What I would expect is, that a geocoder gives me back 
> some kind of answer for every question I ask, being it an exact match on the 
> city or on the street, or only a „similar“ match. It should tell me if there 
> was no 100%-match, there were several records found, matching my street or my 
> city from e.g. 80% to 50%. So then I can decide later on which records I 
> consider a match and which not. In any case the first row returned should be 
> the best match available.
> 
> So I have a couple of questions here: 
> 
> Does anybody know of a geocoder for OSM-data that does this already? 
> I found besides Nominatim there are several other geocoders. But I cannot 
> test them all. Maybe some work already this way.

As a rule of thumb, the elastic-search-based geocoders do a bit better
for misspelled terms but they are still not ideal because elastic search
is optimised for free text, which has a different distribution of words
than addresses.

> There is a Postgresql-module that seems to do just what I want: pg_trgm. It 
> does not seem like Nominatim uses that right now.
> Is there anybody already working on implementing this (or anything similar)?

Trigrams only work with misspellings of a letter or two, they fail
completely when trying to match up abbreviations.

> If not, I would be willing to invest further time and effort into this, but I 
> need some help on the internals of Nominatim, which I’m not firm with. 
> Where would be the right place to integrate this into Nominatim? 
> Does it make sense to try to put this into Nominatim?
> Or would it be easier to use just osm2psql and build on top of that a new 
> query-interface?

One of the most promising new approaches might be libpostal:
https://github.com/openvenues/libpostal

It's not a geocoder but a library for normalising addresses.
So you would use it to preprocess your address and then geocode
the results with a conventional geocoder. There is a php
library for it, so it would be easy to extend the Nominatim
query interface. Although I would probably rather try photon
as the geocoding backend as it will likely catch a few more
spelling errors.

In any case, I'd be very interested in the results if you
experiment with libpostal and would be happy to take a
pull request for Nominatim.

Kind regards

Sarah



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Changeset metadata timestamps out of sync?

2016-08-15 Thread Sarah Hoffmann
Hi,

On Mon, Aug 15, 2016 at 09:10:46PM +0200, Martin Raifer wrote:
> > Martin mentioned a deviation of exactly 4 seconds
> 
> Well, actually, the 4 seconds were only for the one example changeset.
> In other changesets, I saw different time differences (most were even
> smaller if I recall it correctly).

The reason for these time differences is that the clocks on some of
the API frontend servers were out of sync. TomH has fixed the
problem in the meantime.

Only the changeset timestamps were affected. The timestamps for the
objects should be okay as they are created on the backends[*]. So you
should rather trust those when in doubt.

[*] At least when using diff uploads which every decent editor is
doing nowadays.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] New release Nominatim 2.5.0

2016-02-21 Thread Sarah Hoffmann
Hi,

we are happy to announce a new stable release 2.5.0 of the
Nominatim search engine.

This release brings a major overhaul of the built-in web interface.
There is now a web view for reverse geocoding and the details page
has been made more readable. All is now ready for mobile use. 

The second major change is the addition of a number of new functions
for the API:

 - reverse lookup now can use Tiger house number data in the US
 - new feature to return simplified polygons
 - new lookup endpoint that allows to look up address information
   for OSM objects by id
 - new namedetails and extratags parameters to get a full list of
   names or additional tags for each object (requires postgresql >= 9.3)

Other changes include updates for the Tiger data import script, new
configure options for use behind HTTP proxies and vagrant
scripts for running Nominatim in a Ubuntu or CentOS virtual box.

The release is already live on http://nominatim.osm.org.
For more information, including update instructions, see the release
notes at https://github.com/twain47/Nominatim/releases/tag/v.2.5.0

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] [overpass] Re: Minute replication hiccup

2016-02-13 Thread Sarah Hoffmann
On Fri, Feb 12, 2016 at 12:50:32PM +, Tom Hughes wrote:
> On 12/02/16 12:17, Roland Olbricht wrote:
> 
> >something strange has happened to
> >http://planet.osm.org/replication/minute/001/788/263.osc.gz
> 
> Yes, exactly the same thing as the last N times you asked ;-)
> 
> The machine crashed, and because osmosis doesn't fsync the state
> file the last state was lost and Matt had to reset it.

Wouldn't it be better to leave an empty diff file for the broken
sequence id and reset so that it continues with the subsequent id?
Most clients should be able to cope with replayed data and
hopefully also with empty diff files.
In any case, even a crashing client is preferable to silently
loosing data.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Doubts on Nominatim

2015-10-20 Thread Sarah Hoffmann
Hi,

On Tue, Oct 20, 2015 at 07:18:27PM +0530, Logesh Mohanasundaram wrote:
> I have tried to setup Nominatim in own server and I have just tried to import 
> data of “Alberta, canada” from geofabrik and I ended up with error of “dense 
> node cache” and I increased the machine size to 4GB RAM and 60GB storage and 
> I got it working but it took sometime(around an hour or so, not sure) to 
> import the data. I wanted to import the whole North America data in the 
> server. So, what would be the machine size that I would require to set that 
> up and how long will it take to import all the data?

For North America I would recommend a machine with at least 16GB of RAM
and 250GB storage (+ 150GB more if you intend to import the Tiger data).
This should allow you to finish the import in average time (a few days,
depending on the CPU power). If you want to speed that up even more, go
for 32GB of RAM and SSD storage. The Nominatim import is very heavy on
IO and particularly the latter makes a huge difference.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] osm2pgsql Lua tag transformations

2015-06-25 Thread Sarah Hoffmann
On Thu, Jun 25, 2015 at 11:06:41AM +0200, nebulon42 wrote:
 Thank you for explaining this. Would you see the possibility of
 removing relation members from the geometry as something that could
 be added as feature? Or would it be difficult/not worth/...?
 Otherwise I could follow up on this on the osm2pgsql issue tracker.

I don't know how useful it would be for the wider audience but
it would not be very difficult to include. You can open a feature
request on github but seeing that osm2pgsql is best described as
pityware(*) these days, I doubt that much will happen unless such
a request is accompanied by a pull request.

(*) Pretty much abandonend but occasionally somebody takes pity
and fixes the worst issues.

Sarah

 For completeness reasons: Does anybody know of another or possible
 way of solving this problem of platforms being part of the route
 (like here:
 https://www.openstreetmap.org/#map=18/48.19117/16.34794layers=T)?
 
 Thanks, nebulon42
 
 Am 2015-06-24 um 21:37 schrieb Sarah Hoffmann:
 Hi,
 
 On Sat, Jun 20, 2015 at 05:47:43PM +0200, nebulon42 wrote:
 I'm currently experimenting with Lua tag transformations when
 importing data with osm2pgsql and trying to remove members of
 type=route relations (public transport routes) that have the role
 platform from planet_osm_line. The problem I'm trying to avoid can
 be seen here:
 https://www.openstreetmap.org/#map=18/48.19117/16.34794layers=T
 (platforms coloured like routes)
 
 I thought that when I set membersuperseded = 1 for the corresponding
 relation members in filter_tags_relation_member this would work, but
 apparently it isn't.
 
 membersuperseded doesn't really do what you think it does. If a way
 in a relation is marked as superseded, then the way itself will no
 longer be considered an object of its own but just a part of the
 relation. Or to put it the other way around, if membersuperseded is 0
 then the way may still be included in the output tables as a
 separate object (naturally depending on if it has interesting
 tags or not).
 
 It is currently not possible to exclude specific members in a
 relation from taking part in the computation of the final geometry.
 The geometry computation always uses all ways available.
 
 Sarah
 

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] osm2pgsql Lua tag transformations

2015-06-24 Thread Sarah Hoffmann
Hi,

On Sat, Jun 20, 2015 at 05:47:43PM +0200, nebulon42 wrote:
 I'm currently experimenting with Lua tag transformations when
 importing data with osm2pgsql and trying to remove members of
 type=route relations (public transport routes) that have the role
 platform from planet_osm_line. The problem I'm trying to avoid can
 be seen here:
 https://www.openstreetmap.org/#map=18/48.19117/16.34794layers=T
 (platforms coloured like routes)
 
 I thought that when I set membersuperseded = 1 for the corresponding
 relation members in filter_tags_relation_member this would work, but
 apparently it isn't.

membersuperseded doesn't really do what you think it does. If a way
in a relation is marked as superseded, then the way itself will no
longer be considered an object of its own but just a part of the
relation. Or to put it the other way around, if membersuperseded is 0
then the way may still be included in the output tables as a
separate object (naturally depending on if it has interesting
tags or not).

It is currently not possible to exclude specific members in a
relation from taking part in the computation of the final geometry.
The geometry computation always uses all ways available.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] nominatim 'boundingbox' field. Recently changed?

2015-05-21 Thread Sarah Hoffmann
On Thu, May 21, 2015 at 05:00:48PM +, Harry Wood wrote:
 I had a bit of code which was using the boundingbox field of the nominatim 
 response
 
 I'd swear something seemed to change a couple of weeks ago. It now returns a 
 zero sized boundingbox, where previously it returned something more useful. 
 e.g. for any suburb match:
 http://open.mapquestapi.com/nominatim/v1/search.php?format=jsonq=Herne+Hill,+London
 
 My code's pointed at mapquest, but I see the same problem on osm's instance.
 http://nominatim.openstreetmap.org/search?format=jsonq=Herne+Hill,+London
 
 Still getting a sensible bounding box for cities and for ways and various 
 other things, but these suburbs used to be working. Anyone else noticed this 
 change? I guess I'll work around it ...unless it's an easy one for nominatim 
 folks to fix somehow.

I've simplified the bbox computation a few weeks ago which uncovered
a three year old bug with the bbox output code. I've pushed a fix now which
should go live on osm.org in a day or two.

Cheers

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim 2.3 release announcement

2014-10-05 Thread Sarah Hoffmann
Hi,

we are happy to announce the release of a new stable version 2.3 of
Nominatim, the OSM search engine.

There are two new features that are most notable for users of the
search engine: there is now support for waterway relations, which improves
searching for rivers, and POIs like shops and restaurants now
can inherit their address from the building they are in. 

Next to that a lot of smaller bugs have been fixed and further tweaks
to the search algorithm allow for improved ordering of results. This
release also marks the arrival of a functional test suite for
Nominatim.

The new release is available at

http://www.nominatim.org/release/Nominatim-2.3.0.tar.bz2

Installation instructions are at the usual place at

http://wiki.openstreetmap.org/wiki/Nominatim/Installation

You can always try the latest version of Nominatim on

http://nominatim.openstreetmap.org

Regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim 2.3 release announcement

2014-10-05 Thread Sarah Hoffmann
Hi,

On Sun, Oct 05, 2014 at 11:03:39AM +0200, Christoph Hormann wrote:
 On Sunday 05 October 2014, Sarah Hoffmann wrote:
 
  There are two new features that are most notable for users of the
  search engine: there is now support for waterway relations, which
  improves searching for rivers, [...]
 
 Nice.
 
 It seems though they are reported in the same way in the results lists 
 as river lines with 'River' which makes them hard to identify.  

Word of warning: although the instance at nominatim.osm.org (and consequently
the search at osm.org) runs the latest code of Nominatim, it hasn't
been reimported since waterwat relations have been introduced. The
database has only changed to the new schema, where OSM object have
been edited since.

 In principle i would have four major classes for water features:
 
 * standing water areas (natural=water + water!=river/channel)
 * flowing water areas (waterway=riverbank or natural=water + 
 water=river/channel)
 * waterways (waterway tag but not waterway=riverbank)
 * waterway relations

Well, the idea was to make only the 'river line' of rivers searchable
and complete ignore the areas as they have only limited meanining from
a geolocation point of view. So, it should show only waterway relations,
waterway ways that are not part of a relation (and are not something
potentially interesting like a waterfall) as well as natural=water.
Of course, it's all heuristics because tagging is far from consistent.
For some disussions see https://github.com/twain47/Nominatim/issues/155

I'm dimmly aware that there is a new proposal for water areas aiming
to unify fowing and standing water areas. I can't say I'm very happy
with it from a data processing point of view because it mixes primary
geographic objects that are of interest on its own (lakes etc.) with
secondary objects that only describe the area of another object
(i.e. river and riverline). Having to use a second tag to figure out
the meaning of the first is always problematic. Therefore, Nominatim
ignores the whole water=* issue in the hope that the tagging schema
doesn't really catch on. ;)

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim geocoding for Japanese address

2014-04-10 Thread Sarah Hoffmann
On Thu, Apr 10, 2014 at 09:09:21AM +0300, Jukka Rahkonen wrote:
 I have tried to help a user with an address related task on OSM
 forum http://forum.openstreetmap.org/viewtopic.php?id=24757.
 
 I found it quite hard to work with the admin boundary relations of
 OSM so I finally took the municipality borders as old fashion GIS
 polygons from the National Land Survey of Finland and after that
 sorting the unique streetnames by municipalities was very simple to
 do with a spatial SQL query.
 
 As you work with Nominatim, don't you ever feel that there should be
 a separate OSM repository for administrative polygons?

This question comes up from time to time but the idea eventually
always gets rejected because we still don't have a general concept
how to work with multiple databases.

What I'd like to see is someone providing regular dumps of admin
boundaries as geojson or shapefiles, just like dumps are provided
for coastlines.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim geocoding for Japanese address

2014-04-09 Thread Sarah Hoffmann
Hi,

On Wed, Apr 09, 2014 at 12:52:49AM +0900, Satoshi IIDA wrote:
  Japanese address system
 I made Japanese address structure for OSM, last year.
 And got consensus between Japanese mappers. also reported in Tagging ML.
 https://lists.openstreetmap.org/pipermail/tagging/2013-September/014816.html

I rememeber this mail but obviously misunderstood that it is an
finished proposal. I also cannot find much data that follows the schema.
Taginfo reports that there are more addr:block_number than addr:blocknumber
but that might be unrelated.

 Documentation is written in Japanese, but certainly need more easy
 description.
 http://wiki.openstreetmap.org/wiki/JA:Key:addr#.E6.97.A5.E6.9C.AC.E3.81.AE.E4.BD.8F.E6.89.80.E8.A1.A8.E8.A8.98.E3.81.AB.E3.81.A4.E3.81.84.E3.81.A6
 
 Block structure detail (too complex, sorry)
 http://wiki.openstreetmap.org/wiki/JA:Addresses

It's easy enough to understand but it would be really helpful if
you could translate the page to English. I fear that the translation
with Google is not accurate enough.

 Problem is the absence of address data on Japanese area.
 As my early mail, most of admin boundary relation is broken.

Indeed, that makes it more difficult bu I believe that many
place nodes are already there and they are good enough to start with.

 But if we could do so detailed search on OSM, I'm very happy to encourage
 to collect address data on Japan :)

I don't think it will be difficult to implement this schema in Nominatim.
I would kindly suggest that you translate the wiki page, then, if you
have not done that yet, enter the data for one or two of these lower 
neighbourhoods into OSM and then again send a mail to tagging@ for 
discussion (and maybe open a trac ticket for Nominatim at the same
time, I do notmally not follow tagging@). There are some details which
are not clear but it is probably easier to discuss on a concrete example.
Also, there are other parts in the world that use block-based addressing
and it would be great if the tagging can be used there as well.

Sarah


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim geocoding for Japanese address

2014-04-09 Thread Sarah Hoffmann
On Wed, Apr 09, 2014 at 02:23:54PM -0700, Troy Wu wrote:
 
 On Apr 9, 2014, at 11:23 AM, Sarah Hoffmann wrote:
 
  On Wed, Apr 09, 2014 at 12:52:49AM +0900, Satoshi IIDA wrote:
  Japanese address system
  I made Japanese address structure for OSM, last year.
  And got consensus between Japanese mappers. also reported in Tagging ML.
  https://lists.openstreetmap.org/pipermail/tagging/2013-September/014816.html
  
  I rememeber this mail but obviously misunderstood that it is an
  finished proposal. I also cannot find much data that follows the schema.
  Taginfo reports that there are more addr:block_number than addr:blocknumber
  but that might be unrelated.
 
 And then what about other countries?  In Taiwan, there's the idea of main 
 road, alley/subroad, then building identifier.  I'm sure other places will 
 have different address formats.  Someone just mentioned that the best 
 solution is to create a geocoder for each type of address?  Is that the 
 trajectory of geocoding for OSM?  To have a per-country/region based geocoder?

I sincerily hope not.

The reason that geocoding works so badly outside Europe is first of
all a data problem. The only systematic addressing proposal we have
to date is the Karlsruhe schema and that was indeed developed with
European street addressing in mind. There is no use in developing
a localized geocoder until the tagging problem for other addressing
schemas is sorted out.

So if your country addressing doesn't work, by all means, propose
extensions to the current addressing schema that solve your problem.
You should discuss these things in your local community first, then
document it in the wiki and put it out on tagging@ for discussion
to find out if maybe other countries have a similar problem which
can be solved at the same time. And please, open a trac ticket
for Nominatim to let the developers know why addressing in your
country does not work and what you are planning to tag.

Of course, there is no gurantee if/when things get actually implemented
because there are simply too few developers to implement the stuff.
Again, writing a new geocoder won't really help with that problem.
It would be much better if the few people interested in this topic
would join forces and manage to work on one common project.

 And, someone mentioned that Nominatim isn't good in the US?  I'm trying to 
 implement some basic geocoding (all US addresses), and that would be somewhat 
 devastating.  Is this actually the case, before I spend time going down this 
 path?

I'm looking at this from another continent, so I might be wrong,
but the problems with geocoding in the US look mostly like a data
problem as well. Boundary data in particular is really bad in the US.
The good news is that the US community has worked hard on improving
the data in the last year and I'm quite confident that these things
will get sorted out soon. There are also some particularities with
US addressing where the tagging still needs to be defined better (i.e.
postal towns) but apart from that I don't think Nominatim would do
that badly.

There is also now the address repository that Ian Deeds announced a
few days ago. This data could be mixed into Nominatim as external
data quite easily and further improve the rather patchy house number
data.

If all that in its current state is enough for your needs, I don't
know. You should run a trail with a small set of data on the
publicly available instances to find out.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim geocoding for Japanese address

2014-04-08 Thread Sarah Hoffmann
Hi,

On Mon, Apr 07, 2014 at 02:03:19PM +0900, Satoshi IIDA wrote:
 Although I'm not a Nominatim guy :)

I am. :) I'll have a look at the data and try to answer the
trac ticket later. Just some quick response about addresses here.

 3. place name? address?
 Your dataset seems about place/block name rather than address.
 
 As you know, Japanese addressing is city block based. Not street based.
 So perhaps it would need some enhancement on Nominatim.
 (from my memory, block based search is already implemented.)

Nominatim is now able to handle 'street-less' addresses but I'm not sure
that this covers addresses in Japan adequately. There is no problem to
extend Nominatim with a specific block-based addressing schema.
The main isssue is that it is not quite clear if there ever has been a 
consensus about the tagging of block-based addresses. As all the developers
of Nominatim are based in countries with street-based addressing schemas, 
we really need to rely on you there.

So it would be great if you could discuss in your community the tagging
again and find a consensus if not yet has been reached, document that
on a wiki-page and file a trac ticket for Nominatim with a link to that
wiki page and some example areas where the schema is already being used.

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] nominatim / windows

2014-01-21 Thread Sarah Hoffmann
On Tue, Jan 21, 2014 at 11:23:58AM +0100, Dominik Perpeet wrote:
 Thanks for the pointers to the relevant code sections. I don't think the
 error is there, though:
 
 - The script doesn't seem to abort, since in the code pretty much the
 last action is to create the table wikipedia_redirect, which was indeed
 created in my setup.
 - placex and place have their triggers and are not empty

That's good then. There was no CREATE TRIGGER message in your output.

What should happen is that the data gets copied from place to placex.
The trigger of placex computes some additional values for the objects
and, most importantly, sets the value of indexed_status to 1. Only the
rows that have this indexed status will later be reindexed (the step
which reports processing 0 objects in your logs).

 Some other step seems to fail silently, any ideas?

The placex table looks a bit small. I'd expect the number of rows to be in
the same order of magnitude as place. So I wonder if the data loading
happens at all. There is certainly no mentioning of it in the logs.
Try to list the row counts by class/type column.
The numbers should be mostly the same for place and placex. If placex
only contains postcodes and some state boundaries (both are imported
from a different source than the place table)   then the data loading
surely failed.

Sarah


  Looking at the table sizes, it looks like loading the data[1] does not work
  properly. There should be some INSERT triggers on placex which seem to be
  missing. They should be set from an SQL script around here[2].
 
  My guess would be that your version of psql immediately stops the scripts
  when it encounters an error while the unix version just skips over them.
  There are some expected fails which you can also see in your logs:
 
  snip
  FEHLER:  Typ A»place_boundingboxA« existiert nicht
  FEHLER:  Typ A»place_boundingboxA« existiert nicht
  FEHLER:  Typ A»place_boundingboxA« existiert nicht
  LINE 4:   result place_boundingbox;
  snap
 
  It should really just ignore them and continue.
 
  Sarah
 
 
  [1] https://github.com/twain47/Nominatim/blob/master/utils/setup.php#L377
  [2] https://github.com/twain47/Nominatim/blob/master/sql/tables.sql#L224
 

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] nominatim / windows

2014-01-20 Thread Sarah Hoffmann
Hi,

On Mon, Jan 20, 2014 at 09:02:23PM +0100, Dominik Perpeet wrote:
 Some may remember me from previous messages (a long time ago) about
 osm2pgsql on Windows. I've gone a step further this time and compiled
 nominatim as well and ran it all using a modified wamp.
 
 Things seem to work on the surface, but for some reason nominatim isn't
 really generating any data. I've used pretty much the latest version
 available of osm2pgsql and nominatim.
 
 Any hints or thoughts?
 
 I have uploaded more extensive logs to http://customdebug.com/osm/nominatim/
 (database messages, general messages and table lenghts [row counts])

Looking at the table sizes, it looks like loading the data[1] does not work
properly. There should be some INSERT triggers on placex which seem to be
missing. They should be set from an SQL script around here[2].

My guess would be that your version of psql immediately stops the scripts
when it encounters an error while the unix version just skips over them.
There are some expected fails which you can also see in your logs:

snip
FEHLER:  Typ A»place_boundingboxA« existiert nicht
FEHLER:  Typ A»place_boundingboxA« existiert nicht
FEHLER:  Typ A»place_boundingboxA« existiert nicht
LINE 4:   result place_boundingbox;
snap

It should really just ignore them and continue.

Sarah


[1] https://github.com/twain47/Nominatim/blob/master/utils/setup.php#L377
[2] https://github.com/twain47/Nominatim/blob/master/sql/tables.sql#L224

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

2013-11-06 Thread Sarah Hoffmann
Hi,

[another try, this time to the list]

On Wed, Nov 06, 2013 at 10:10:03AM +0100, Frederik Ramm wrote:
 On 07/13/2013 01:03 PM, Jocelyn Jaubert wrote:
  I'm currently using a workaround, which is to set maxInterval in
  configuration.txt to 0.5 day to make sure that osmosis is not trying to
  merge two diffs. Then I had to add a loop to run osmosis until there are
  no recent diffs.
 
 I've just been bitten by the same problem when trying to keep an
 Europe-only Nominatim instance up to date with Geofabrik diffs.
 
 I wonder if simply dropping the --simplify-change from Nominatim's
 osmosis invocation will solve things - might lose a little performance
 for things modified on subsequent days but hey. (One could even do this:
 try with --simplify-change by default but if this fails, re-run
 without... Nominatim already re-runs osmosis if it fails so I'd just
 have to add the drop simplify-change code.)

That does not work either because osm2pgsql chokes on a osc diff
that was not simplified.

  Maybe the best way to fix this is to add a note on a website explaining
  that the diffs generated by Geofabrik are not meant for direct
  consumption by osmosis with a big value on maxInterval ?
 
 I'll do that.

With Nominatim 2.1 and the settings as recommended at the Nominatim
installation website [1], the update process should be doing the
right thing and apply the daily diffs one after another.

Sarah

[1] 
http://wiki.openstreetmap.org/wiki/Nominatim/Installation#Setting_up_the_update_process

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev


Re: [osmosis-dev] (no subject)

2013-08-25 Thread Sarah Hoffmann
Hi,

are you sure that you got the right mailing list?

On Sun, Aug 25, 2013 at 09:11:54PM +0100, Simon Nuttall wrote:
 I've hit this issue about 48 hours into processing a Europe - wide
 nominatim. Is there a way of proceeding?

It seems to be just an update, simply run it again. --import-osmosis-all
will try to reapply the change automatically when restarted after you've
fixed the underlying problem.

 25-Aug-2013 19:45:34 org.openstreetmap.osmosis.core.Osmosis run
 INFO: Total execution time: 39209229 milliseconds.
 Completed for 2013-06-29T19:59:05Z in 1361.3 minutes
 /home/nominatim/Nominatim/osm2pgsql/osm2pgsql -klas -C 2000 -O
 gazetteer -d nominatim
 /home/nominatim/Nominatim/data/osmosischange.osc
 Using projection SRS 4326 (Latlong)
 Allocating memory for dense node cache
 Allocating dense node cache in one big chunk
 Allocating memory for sparse node cache
 Sharing dense sparse
 Node-cache: cache=2000MB, maxblocks=256001*8192, allocation method=11
 Mid: pgsql, scale=1000 cache=2000
 Setting up table: planet_osm_nodes
 Setting up table: planet_osm_ways
 Setting up table: planet_osm_rels
 
 Reading in file: /home/nominatim/Nominatim/data/osmosischange.osc
 COPY_END for place failed: ERROR:  column addr_place of relation
 placex does not exist
 LINE 2:   street, addr_place, isin, postcode, country_code, extr...

This looks like you have updated your Nominatim version without migrating
the database. There have been some changes in the DB schema recently.
The migration for the addr_place column is described here:
https://github.com/twain47/Nominatim/pull/54#issuecomment-17277092
(needed after ddc46acd26)

The other significant DB schema changes was:
https://github.com/twain47/Nominatim/pull/45#issuecomment-16157763
(needed after 23d303124e)

Sarah

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev


[OSM-dev] Reminder of Nominatim usage policy

2013-06-05 Thread Sarah Hoffmann
Hi devs,

the Nominatim search API has seen quite an increase in popularity in
the last months. We are in principle happy to see you experiment
with new and exciting uses of the API. However, there have been a few
applications that have been overusing our servers making it increasingly
difficult to keep up the service. Please be remainded that our services 
are running on hardware made possible through donations and that their 
capacity is limited.

To ensure that your search results will continue to be delivered
promptly, a more ridgid enforcement of the usage policy will be put
in place in the coming weeks. In particular, you should be aware of
the following two points:

The usage limits stated in the usage policy apply on an application base.
That means if you are using Nominatim search services on a website or in
a distributed or mobile application, then all your clients together must
not produce more traffic than the stated usage limits.

Users of the API must include either a customized user agent or a referer.
Clients that fail to send an identification or only send a general-purpose 
user agent will be no longer permitted.

The fully usage policy can be found at
http://wiki.openstreetmap.org/wiki/Nominatim_usage_policy

If you are a user of the Nominatim API, please check again that your 
software complies with the usage policy to avoid bad surprises. If you 
are a heavy user of the API, please look into alternatives like
commercial providers or installing your own instance of Nominatim.

Kind regards

Sarah Hoffmann
(part of the OSM sys-admin team)

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] mod_tile stable version ?

2013-03-28 Thread Sarah Hoffmann
On Thu, Mar 28, 2013 at 07:36:56AM -0700, Kai Krueger wrote:
 Despite possibly being the person who has recently been most active in
 committing code to mod_tile / renderd and osm2pgsql, I have been reluctant
 to claim the official maintainer title, as I didn't want to be the gate
 keeper or bottleneck. But perhaps I should just take that responsibility, or
 at least make sure to try and be more responsive to patches.

I'd be very happy to see you step up as maintainer and move the stuff
to github. I'd feel much more comfortable to submit pull requests to 
github instead of doing direct SVN commits that nobody even knows 
about unless the code fails in some annoying way.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Data: DB Error: no such table

2013-02-12 Thread Sarah Hoffmann
On Tue, Feb 12, 2013 at 11:54:16AM +, Steven Walsh wrote:
 Hello,
 
 I'm trying to build nominatim but found this error when it as complete: Data:
 DB Error: no such table
 
 
 From looking around similar questions it seems that rebuilding is the only
 option but I wanted to ask you if there was anything else that could be
 done as I'd rather not have to wait again. The only thing maybe of interest
 is that nominatim is built on a mounted data partition and I was thinking
 it could just be looking in the wrong places for the database, but I've no
 idea where to look.

It looks like the php script can find a database called 'nominatim' and can
also access it but it is somehow completely empty. Are you sure that the
import process finished successfully? Did you run any additional setup commands
after the import? If you are convinced that the import was ok, you should try 
to 
find out where your data went. Can you access the database from the command 
line (using
psql -d nominatim) and can you see the data in there? Also look out for
multiple versions of postgresql being installed in parallel.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] nominatim-like wildcard search

2012-08-29 Thread Sarah Hoffmann
Hi Per,

this topic seems to come up frequently lately and I'm actually
glad you asked before going off implementing something. 
So forgive me if I jump in with a more general remark on the topic.

On Wed, Aug 29, 2012 at 12:26:02PM +0200, Per Eric Rosén wrote:
 I'm trying to build a autocompletion name lookup for a HTML5-based
 cycle map. Is there any API where you can make wildcard
 nominatim-style lookups?
 
 For exempel Bre* would return Bremen, Brenner etc. I will
 limit my queries by bbox, but other sorting and limiting, like
 sorting by distance to a given point, or place category, would be
 useful.
 
 Probably somewhat like
 http://oegeo.wordpress.com/2010/11/08/a-better-search-field-for-openstreetmap/
 
 Seem to be suggested in
 http://wiki.openstreetmap.org/wiki/Nominatim/Version2
 but I see no indication of that being implemented ...

This is not yet implemented in Nominatim and using the current API
to implement it client-side will not work because Nominatim only
returns complete matches, i.e. in your example 'Bre' will never return
'Bremen'. It will return an empty result and such empty queries are
the really expensive ones that give the server a hard time.

Please, people, *do not implement auto-complete on top of nominatim.osm.org*.
You will just end up flooding the server with so many nonsense requests
that you attract the attention of the sysadmins. You do not want that.

If you run your own installation of Nominatim, you are of course welcome
to do whatever you want with it.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Problem building osm2pgsql from source

2012-06-13 Thread Sarah Hoffmann
On Wed, Jun 13, 2012 at 11:45:50PM +0100, Jon Burgess wrote:
 On Wed, 2012-06-13 at 15:15 -0700, Michael Corey wrote:
  Hi all: Please excuse me if this isn't the right list for this.
  
  
  
  While following the directions for installing osm2pgsql from source, I
  get a fatal error while running 'make':
  
  build_geometry.Tpo -c -o build_geometry.o build_geometry.cpp
  build_geometry.cpp:29:26: fatal error: geos/version.h: No such file or
  directory
  compilation terminated.
  
 What distro are you building this on? 
 
 The version.h file should be part of the geos library headers, e.g. 
 
 Ubuntu:
 $ dpkg -S /usr/include/geos/version.h 
 libgeos-dev: /usr/include/geos/version.h

Also try to install the libgeos++-dev package. 
Debian has moved the headers there recently.

Sarah


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim's reverse geocoding statistics

2012-05-02 Thread Sarah Hoffmann
Hi,

On Wed, May 02, 2012 at 09:56:07AM +0200, Charles DESNEUF wrote:
 We are trying to build our own reverse geocoding service, based on
 Nominatim and we have an issue with it. Nominatim does work but when
 running a benchmark on the reverse geocoding page we are unable to get more
 than 13 or 14 pages per second.
 CPU isn't loaded, Memory is fine and IO are goods...
 I would like to know if some of you tried to benchmark his installation and
 know how many request the service can handle ?

poldi, OSM's Nominatim instance[1], currently tops somewhere around
220 request/s[2], although that includes about 5% search requests which
are much more expensive than calls to reverse. Also keep in mind
that it has the update process running in the background.

datendelphin has done some stress testing on a Hetzer EX-4S[3]
and reported that it manages around 120 requests/s (without any updates
running).

Both machines run a full planet with postgresql configured as described
in the wiki.

 And if you have an idea about what parameters (Php, Postgres, Apache ...)
 can we play on to improve the results I'll be glad to ear it.

Switching off keep-alive in Apache did help quite a bit. 
Also, you might want to switch the IO scheduler to deadline.

For postgresql, it might be worth to rerun VACUUM ANALYZE.

Sarah

[1] http://wiki.openstreetmap.org/wiki/Servers/poldi
[2] http://munin.openstreetmap.org/openstreetmap/poldi.openstreetmap/index.html
[3] http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim Installation problem - nominatim.xml not found

2012-01-30 Thread Sarah Hoffmann
Hi Anwar,

On Mon, Jan 30, 2012 at 10:58:47AM +0700, Anwar Azulfa wrote:
 Hi All,
 
 Long time no continue this project,i have follow instruction from
 http://open.mapquestapi.com/npi/
 
 Now i have finished imported index file and generate web.
 But when i try to query, error happen.I don't know what's that but there is
 permission denied message.
 
 Let see on
 
 http://www.osmosa.net/search/search.php?q=jakartaviewbox=-140.89%2C64.68%2C-20.83%2C31.75
 
 
 What should i do ?

Two things: first of all, in settings/settings.php you need to adapt
the base URL for your server. Find the line with CONST_Website_BaseURL
and set it to http://www.osmosa.net/search/. Second, you need to make
sure that your server redirects /search/search to /search/search.php.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim Installation problem - nominatim.xml not found

2011-12-21 Thread Sarah Hoffmann
Hi,

On Wed, Dec 21, 2011 at 03:16:39PM +0700, Anwar Azulfa wrote:
 Thanks for all your reply,
 
 
 I have re deploy my osm2pgsql with nominatim_commit.patch
 
 But the error still same.http://pastebin.com/ctQScVhA
 
 What will happen if i ignore it ?

As Mike said, it is not a good idea to ignore the error because it
tells you that something more fundamental is going wrong.

  On Wed, Dec 21, 2011 at 06:17:54AM +0700, Anwar Azulfa wrote:
   3.load planet data
   ./osm2pgsql -S default.style --slim -d gazetteer -C 2048
   ~/AllMap/indonesia.osm.bz2

Actually, you are using the wrong output here. You
are creating a database for mapnik, not for the gazetteer. The
correct command line should be:

./osm2pgsql -lsc -d gazetteer -O gazetteer -C 2048 ~/AllMap/indonesia.osm.bz2


Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Nominatim Installation problem - nominatim.xml not found

2011-12-20 Thread Sarah Hoffmann
Hi,

On Wed, Dec 21, 2011 at 06:17:54AM +0700, Anwar Azulfa wrote:
 Now i have user gazetteer from
 http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/gazetteer
 
 this is my step by step installation which i follow:
 
 1.createdb
 
 2.import sql file
 
 cat /usr/share/postgresql/8.4/contrib/pg_trgm.sql | psql gazetteer
 cat /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql | psql
 gazetteer
 cat /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql |
 psql gazetteer
 
 3.load planet data
 ./osm2pgsql -S default.style --slim -d gazetteer -C 2048
 ~/AllMap/indonesia.osm.bz2
 
 4.copy lib into /usr/local/share
 sudo mkdir /usr/local/share/gazetteer
 sudo cp .libs/gazetteer.so /usr/local/share/gazetteer/gazetteer.so
 
 5.add suplementary data
 
 when i import file below to db
  cat import_country_name.sql | psql gazetteer
 
 i get error message like http://pastebin.com/ctQScVhA

There is currently a bug in osm2pgsql. It forgets to actually commit
your data. I'm trying to get that fixed in svn. 

For the moment, I have a patch attached. Apply it, recompile osm2pgsql and 
redo everything from step 3.

Sarah
Index: output-gazetteer.c
===
--- output-gazetteer.c  (revision 27287)
+++ output-gazetteer.c  (working copy)
@@ -1020,6 +1020,7 @@
 //   Options-mid-iterate_relations( gazetteer_process_relation );
 
/* No longer need to access middle layer */
+   Options-mid-commit();
Options-mid-stop();
 
/* Stop any active copy */

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Harmless edits (was: Change in wtfe.gryph.de Quick History Service API)

2011-12-03 Thread Sarah Hoffmann
On Sat, Dec 03, 2011 at 01:13:14AM +0100, Frederik Ramm wrote:
 Everyone is invited to play with this script and see what happens. I
 plan to make this the basis of the v2 WTFE service, meaning that in
 the future editors will likely *not* highlight stuff that my script
 deems harmless.
 
 Here's the - hacky, perly - source code: http://wtfe.gryph.de/harmless.pl
 
 Please don't do mass evaluations with this web service, as it runs a
 history query against the API in the backend and this is quite
 costly. If you want to run this on a large area, download the .pl
 file and make yourself a full history extract with Peter Koerner's
 history splitter, then run the perl script on the XML. It can
 process anything up to the complete planet if you have the patience.

Just a reminder on the side: Simon Poole provides history extracts for 
some countries here: http://odbl.poole.ch/extracts/ They are softcut
using the polygons from Geofabrik. Might come in handy here.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Harmless edits (was: Change in wtfe.gryph.de Quick History Service API)

2011-12-03 Thread Sarah Hoffmann
Hi,

On Sat, Dec 03, 2011 at 01:13:14AM +0100, Frederik Ramm wrote:
I have finalized a script that can analyze an object's history
 and determine if certain edits are non-edits (i.e. nothing of note
 was changed at all), or harmess (i.e. the object was changed and
 might have to be rolled back if the contributor does not agree to
 the license change, but the rollback will likely not affect the
 quality much).
 
 [...]
 
 You can try out my script here, by adding a way/node/relation id to
 the URL like so:
 
 http://wtfe.gryph.de/harmless/way/40103577
 
 The output is a break-down of what my script thinks has happened to
 the object, and which edits are zero-edits (severity: 0) or
 harmless (severity: 1). After the version analysis, it summarizes
 the user contributions - each user is afforded the highest severity
 of all his changes.

There is a small oddity: when a non-agreeing user deleted an object
then the script notes that down as a zero-edit and ignores the fact
that the object is gone. Example:

http://wtfe.gryph.de/harmless/node/1300187843

see history: http://www.openstreetmap.org/browse/node/1300187843/history

Might happen only if there are no tags.

I'm also slightly confused by the user summary. What do 'relevant change'
and 'no change' mean? I would have expected 'relevant change' to be only
those that are non-zero edits by non-agreers but that does not seem to be
the case.


Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Ways

2011-08-12 Thread Sarah Hoffmann
Hi,

I recently stumbled upon another strange case that may cause trouble:
a way where two consequetive nodes had different IDs but
exactly the same coordinates.

Sarah

On Thu, Aug 11, 2011 at 07:07:34PM +0200, ant wrote:
 I'm currently writing processing software and a couple of questions
 regarding ways have come up. None of the following examples makes
 practical sense, and I believe (hope) that the editors prevent such
 things from happening. But what about ...
 
 ... subsequent duplicate references?
 
 way
   nd ref=1/
   nd ref=2/
   nd ref=2/
   nd ref=3/
 /way
 
 ... closed ways trivial enough not to represent an area?
 
 way
   nd ref=1/
   nd ref=2/
   nd ref=1/
 /way
 
 way
   nd ref=1/
   nd ref=1/
 /way
 
 
 The reason why it bothers me is that I feel my program should deal
 with these cases. So the technical part of my question is: How does
 the API handle it? Which of the above should I expect to be found in
 the planet file, which not?
 
 cheers
 ant
 
 ___
 dev mailing list
 dev@openstreetmap.org
 http://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Converting OSM Mapnik stylesheet to Cascadenik or Carto

2011-07-30 Thread Sarah Hoffmann
On Sat, Jul 30, 2011 at 06:21:51PM +0200, Igor Brejc wrote:
 Great work and thanks for the detailed info.
 
 One comment/question: from your description (using Mapnik) and from looking
 at the json_getter.py I conclude you rely Osm2pgsql DB schema. I mentioned
 earlier that I didn't want to limit Maperitive to using this flattened
 schema, but would also try to support the original OSM DB schema (or even
 some other derived schema). The main reason is that IMHO flattening of OSM
 data  produces some loss of information which is vital for creating high
 quality maps.
 
 Example: Osm2pgsql flattens bus route relations into one or more chunks of
 consecutive ways. But in order to render a bus network (like this one
 http://www.harvestersway.co.uk/images/location_busmap.jpg), one would need
 to reconstruct a graph of all the bus routes, which can have several routes
 (relations) share the same way, and each relation can consist of several
 ways. So it's a many-to-many relationship, which would be difficult to
 reconstruct from this flattened schema.
 
 So my question is: how would you go about implementing MapCSS support for an
 original schema, if at all? This is why I expressed concerns about
 performance, since I don't see an efficient way of fetching the data using a
 declarative map styling.

If you want to do something like a graph, you need to do heavy preprocessing
and create your own database schema anyways. It is not too difficult
to do this in a way that suits the renderer. I did something like this for
hiking.lonvia.de. The database behind the routes is a graph of hiking
relations and there is a table that defines the render style for each segment
which are then rendered with a handful of Mapnik rules. I looked into using 
a MapCSS-based renderer for that. The main obstacle with MapCSS was that it 
defines selectors that are very much tied to the osm2pgsql schema.
This could be changed by allowing selectors to be an arbitrary
string. Then you can write a generalized renderer that maps these selector 
strings to table names and the element names to columns. This should allow
to use any customized database schema you want (as well as osm2pgsql)   
as long as there is a well-defined way to get the geometry for a table 
row (and the type of geometry, for that matter).

Only the descendant operator, I would not know how to fit into this schema.
Once you do preprocessing, the notion of relation, ways and nodes in an OSM
sense is mostly discarded.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Schema version problem with osmosis postgresql/postgis

2011-05-31 Thread Sarah Hoffmann
Hi,

On Tue, May 31, 2011 at 03:25:15PM +0200, Benjamin Meier wrote:
 I want to setup a Postgresql/postgis DB and fill it with data from a  
 xml-file using osmosis. For this I'm using this tutorial:
 http://wiki.openstreetmap.org/wiki/Osmosis_PostGIS_Setup

 I'm using Ubuntu 10.10 with Postgresql-8.4 and osmosis-0.39

 I followed the instructions from the tutorial to setup the database and  
 edited some paths to match with my filesystem.

 sudo su - postgres
 createdb osm
 createlang plpgsql osm
 createuser user

 psql -d osm -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
 psql -d osm -f  
 /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
 psql -d osm -f /usr/share/postgresql/8.4/contrib/hstore.sql
 psql -d osm -f  
 /home/benny/Desktop/DA/programme/osmosis-0.39/script/pgsimple_schema_0.6.sql

 Those commands worked without errors.

 When I run osmosis with:
 ./osmosis --read-xml file=/media/daten/osm/someplace.osm --write-pgsql  
 user=user database=osm password=osm

You are mixing two different schemata. If you initialise the database
with pgsimple_schema_0.6.sql, you need to import with '--write-pgsimp'.
For '--write-pgsql' you need to initialise the database with the
pgsnapshot_schema_0.6.sql script. See also
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#PostGIS_Tasks_.28Snapshot_Schema.29

The main difference is that the snapshot schema uses hstore for tags
while pgsimple uses a simple table. I'd recommend using pgsnapshot.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Multipolygon processing (was: osm2spatialite!)

2011-02-18 Thread Sarah Hoffmann
Hi,

On Fri, Feb 18, 2011 at 08:22:54AM +0100, Frederik Ramm wrote:
 Hi,

 On 02/18/11 04:35, Daniel Sabo wrote:
 Makes sense. Do you do anything about touching inner rings?

 No, and I haven't thought of a good way to handle it. If you have an 
 algorithm that works well I'd be interested it it :).

 I use the GEOS library and I test all pairs of inner rings for  
 intersections. Then if I find a pair with an intersection, I check if  
 that intersection is a line (rather than just a point). If it is a line,  
 then I compute the SymDifference between the two and throw the result  
 of that into the polygonizer which hopefully will be able to make one  
 ring from it. You can look at my algorithm here

I have tried a slightly different approach to the multipolygon problem
which goes like this: first, find all possible linear rings by first
joining the ways at the obvious points and then repairing the remaining
gaps. Second, compute the outer hull of all rings, which gives valid
polygons for all rings, i.e it resolves any self-intersections. Finally, 
compute a MultiPolygon by taking the SymDifference of all these polygons.

The SymDifference conveniently sorts out outer and inner rings, and can
handle nested inner rings, multiple outer rings and touching rings. 
Only rings that intersect each other might have to be treated in a 
special way.

It turns out that the tricky part is the computation of the outer hull.
GEOS' buffer() function should be able to do that but it is also produces
wrong results when encountering self-intersections, so there seems to be no
way around writing a hand-crafted sweep algorithm.

 All this is probably only halfway there. I'd be very interested in ideas  
 how to fix broken multipolygons. There is some code there (line 117ff  
 tries to repair self-intersections and 343ff tries to fill gaps in  
 rings) but still OSM users come up with ever more invalid polygons ;)

How about a gallery of spectacularly misshaped multipolygons?

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.

2010-12-02 Thread Sarah Hoffmann
Hi Andrzej, 

On Wed, Dec 01, 2010 at 03:19:18PM +0100, andrzej zaborowski wrote:
 A further comment on splitting a big dataset into areas is that if the
 areas are disjoint (like in the case of countries, provinces and other
 administrative areas) or nearly disjoint (like in the case of their
 bboxes) you can build a sort of look-up tree from the list of bboxes,
 so that with n bboxes you only need a little more than log n is this
 node inside? checks.  So for example if you're splitting the planet
 into 2 areas, you only need about 15 tests for every node by doing
 a sort of bisection search.

Do you know if there already exists an implementation that creates such 
a look-up tree from the administrative boundaries in the OSM DB?


Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Is there a way to use simple schema without hstore

2010-11-21 Thread Sarah Hoffmann
On Sat, Nov 20, 2010 at 11:52:58AM +1100, Brett Henderson wrote:
 On Fri, Nov 19, 2010 at 10:50 PM, Sarah Hoffmann lon...@denofr.de wrote:
 
  On Fri, Nov 19, 2010 at 09:37:33AM +0100, Andreas Kalsch wrote:
   If you're applying diffs to the database you can enhance the
   osmosisUpdate() function (initially empty, but can be customised) to
   keep your separate tags tables up to date during each diff
   application.  You will need to run the
   pgsql_simple_schema_0.6_action.sql script against the database so
   that all actions during a diff are logged and can be used by your
   osmosisUpdate function to know which records need to be re-processed.
   Is it possible to truncate the actions table for myself so that a
   separate script can access the changes?
 
  Simply copy away the information from the action table somewhere
  persistent in the osmosisUpdate function. Works fine.
 
  However, +1 from me for an action table that can be truncated manually.
 
 
 Is there likely to be a noticeable performance improvement in doing this?

I doubt that. Compared to the entire update task, the overhead of copying
is negligible. It is more a design question. I prefer to keep osmosis and
the scripts for derived tables strictly apart. Doing part of the update
process for derived tables in the osmosisUpdate function intermangles the
two and is very difficult to debug. What was the idea behind osmosisUpdate?
To allow the code to be executed within the same transaction as the changeset
application?

 My preference if the overhead is small would be to add a contrib script to
 Osmosis that installs a non-truncating table that is updated by
 osmosisUpdate, and a customised osmosisUpdate function.  It keeps the pgsql
 tasks simpler if I can do that.

I would have expected that an implementation without update function and a
persistent action table is simpler. Or do you mean, providing both variants
is too complex? In that case, don't worry about it. The current osmosisUpdate
does what I need and writing an apropriate function is simple. I'll just no 
longer think of it as a quick and dirty hack but as the proper way to do 
it. ;)

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Is there a way to use simple schema without hstore

2010-11-19 Thread Sarah Hoffmann
On Fri, Nov 19, 2010 at 09:37:33AM +0100, Andreas Kalsch wrote:
 If you're applying diffs to the database you can enhance the 
 osmosisUpdate() function (initially empty, but can be customised) to 
 keep your separate tags tables up to date during each diff  
 application.  You will need to run the 
 pgsql_simple_schema_0.6_action.sql script against the database so 
 that all actions during a diff are logged and can be used by your 
 osmosisUpdate function to know which records need to be re-processed.
 Is it possible to truncate the actions table for myself so that a 
 separate script can access the changes?

Simply copy away the information from the action table somewhere
persistent in the osmosisUpdate function. Works fine.

However, +1 from me for an action table that can be truncated manually.

 The older Osmosis 0.36 is still available so you don't have to upgrade. 
  It remains compatible with 0.6 XML files.  Finally, if there is enough 
 demand for the older schema style the old tasks can be pulled back out 
 of SVN and run alongside the new ones, but I'm not keen to do that 
 without good reason.  I did consider trying to support both styles of 
 table in the same tasks by dynamically detecting what tables are 
 installed, but it increases the code complexity considerably and I 
 didn't think the effort was worthwhile.
 *) With that, you would provide a downward compatible solution that I would   
 appreciate a lot!

I haven't tried yet but it should be fairly simple to create a view that
simulates the old tag tables. Probably not the best solution performance-
wise but useful for a gradual update of old scripts.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 'missing column in osmosis pgsql dump' - revisited from June 2009

2010-05-14 Thread Sarah Hoffmann
On Fri, May 14, 2010 at 09:58:14AM +1000, Brett Henderson wrote:
 On Fri, May 14, 2010 at 9:27 AM, Middle Fork GIS 
 middlefork...@gmail.comwrote:
 
   The 0.31 release will be fine, but the latest subversion code should
  also be stable.
 
  I seem to have rediscovered this problem, and it seems to exist in the with
  the 0.31 version but be fixed in the svn snapshot from 5/11/2010:
 
 
 Hi Steve,
 
 Is there any reason you're still using 0.31?  0.35 has been released and
 will include these fixes so you shouldn't need to compile your own from SVN.

I recently had a version problem because there was an outdated link on the page
http://wiki.openstreetmap.org/wiki/Osmosis/Installation

It pointed to an old version on your own server. I've taken the liberty to 
fix the wiki page now.

BTW I noticed that in the release tarballs bin/osmosis is not executable.
Not a big problem but a bit inconvenient.

Sarah

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


Re: [osmosis-dev] Postgis is in repo1.maven.org

2009-12-10 Thread Sarah Hoffmann
Hi,

On Wed, Dec 9, 2009 at 2:55 AM, Hakan Tandogan hakan at gurkensalat.comwrote:

 Hi Brett,


 I finally got postgis-1.3.2 synched to repo1.maven.org (creating accounts
 at sonatype, signing artifacts and POMs, etc. etc. etc.)

 I can't check the ant resolve part in ivy till I later tonight, could
 you please recheck that part on your machine as well?


I just tried to build osmosis from a newly checked out version and
this does not seem to work for me. I attach the full error message
below. I actually can't find any archive postgis on repo1.maven.org.
There only seems to be a postgis-main. Am I missing something?

Sarah



[ivy:resolve]   commons-logging#commons-logging;1.0.3 by 
[commons-logging#commons-logging;1.1.1] in [test]
-
|  |modules||   artifacts   |
|   conf   | number| search|dwnlded|evicted|| number|dwnlded|
-
|  compile |   16  |   1   |   0   |   1   ||   14  |   0   |
|  default |   19  |   1   |   0   |   1   ||   17  |   0   |
|   test   |   28  |   2   |   0   |   3   ||   24  |   0   |
-
[ivy:resolve] 
[ivy:resolve] :: problems summary ::
[ivy:resolve]  WARNINGS
[ivy:resolve]   module not found: org.postgis#postgis;1.3.2
[ivy:resolve]    local: tried
[ivy:resolve] /home/sarah/.ivy2/local/org.postgis/postgis/1.3.2/ivys/ivy.xml
[ivy:resolve] -- artifact org.postgis#postgis;1.3.2!postgis.jar:
[ivy:resolve] 
/home/sarah/.ivy2/local/org.postgis/postgis/1.3.2/jars/postgis.jar
[ivy:resolve]    shared: tried
[ivy:resolve] 
/home/mapnik/osmosis/repo/org.postgis/postgis/1.3.2/ivys/ivy.xml
[ivy:resolve] -- artifact org.postgis#postgis;1.3.2!postgis.jar:
[ivy:resolve] 
/home/mapnik/osmosis/repo/org.postgis/postgis/1.3.2/jars/postgis.jar
[ivy:resolve]    public: tried
[ivy:resolve] 
http://repo1.maven.org/maven2/org/postgis/postgis/1.3.2/postgis-1.3.2.pom
[ivy:resolve] -- artifact org.postgis#postgis;1.3.2!postgis.jar:
[ivy:resolve] 
http://repo1.maven.org/maven2/org/postgis/postgis/1.3.2/postgis-1.3.2.jar
[ivy:resolve]   ::
[ivy:resolve]   ::  UNRESOLVED DEPENDENCIES ::
[ivy:resolve]   ::
[ivy:resolve]   :: org.postgis#postgis;1.3.2: not found
[ivy:resolve]   ::
[ivy:resolve] 
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/home/mapnik/osmosis/build-ivy.xml:56: impossible to resolve dependencies:
resolve failed - see output for details



___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev