Re: [OSM-dev] Missing records from osmium-tool tags-filter

2019-11-12 Thread Michael Reichert
Hi Craig,

Am 12.11.19 um 15:45 schrieb Craig Kochis via dev:
> Then exporting to JSON
> 
> osmium export poi.osm.pbf --output-format=geojsonseq --omit-rs
> --add-unique-id=type_id -v -o poi.json
> 
> And some records are being omitted that I would expect to be in the output.
> For instance, Yankee Stadium:
> https://www.openstreetmap.org/way/24801630 matches
> the "leisure" and "sport" tags, yet that way isn't in the result set. If I
> run tags-filter with just "leisure" it will show up, so I'm wondering if
> there's something happening by specifying multiple tags that I'm not aware
> of.

If the way does not appear in the final JSON, it could miss because the
OSM file misses one or multiple nodes referenced by the way. Therefore,
building the geometry is not possible. Use "osmium check-refs" to check
the referential integrity of an OSM file. Please mind that the input
file to check-refs needs to be sorted (can be done using the Osmium tool
as well; usually OSM files are sorted).

https://docs.osmcode.org/osmium/latest/osmium-check-refs.html
https://docs.osmcode.org/osmium/latest/osmium-sort.html

Best regards

Michael

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Issue with way id 742619958 / changeset 76711047 from 2019-11-06T15:32:03Z (Invalid layer value)

2019-11-08 Thread Michael Reichert
Hi Bart,

Am 08.11.19 um 17:04 schrieb Bart Smienk:
> I've found out my OSM Tile Server has stopped updating and osm2pgsql
> (v1.2) gives off the following error:
> 
> DB writer thread failed due to ERROR: result COPY_END for
> planet_osm_line failed: ERROR:  invalid input syntax for integer: "-"
> CONTEXT:  COPY planet_osm_line, line 1437, column layer: "-"
> 
> Looking through its changefile, the following entry can be found:
> 
>      uid="978786" user="padvinder" changeset="76711047">
>   
>   
>   
>   
>   
>     
> 
> This seems to be malformed, as the layer should've never been "-", as
> this is not a valid integer.

It is valid from API point of view as any tags are just a pair of
strings (up to 256 characters). The OSM API is living freedom of
tagging, it does care how your keys and values look like except some
basic checks such as length of the strings.

> You can see this for yourself at:
> https://planet.openstreetmap.org/replication/minute/003/747/729.osc.gz
> Also more information on the layer key is over here:
> https://wiki.openstreetmap.org/wiki/Key:layer
> 
> Not sure where to make the actual bug report, so I'm mailing the dev
> mailgroup, does anyone of you know where this issue should be posted so
> this issue won't present itself again in future replication files?

It is an Osm2pgsql bug which should check that valid integers are
written via PostgreSQL COPY into a integer column. The bug tracker is at
https://github.com/openstreetmap/osm2pgsql/issues

I cannot reproduce your bug. I tried the master branch (currently equal
to 1.2.0 release).

This does not lead to the bug you observed:
./osm2pgsql -d osm2pgsqldev \
--style ~/git/openstreetmap-carto/openstreetmap-carto.style \
--tag-transform ~/git/openstreetmap-carto/openstreetmap-carto.lua \
--cache 10 w742619958.osm

And even using C tag transform works:
./osm2pgsql -d osm2pgsqldev \
--style ~/git/openstreetmap-carto/openstreetmap-carto.style \
--cache 10 w742619958.osm

Using the Osm2pgsql style file (and C tag transforms) works as well:
./osm2pgsql -d osm2pgsqldev --style ../default.style --cache 10
/tmp/w742619958.osm

The input file is (without personal metadata which does not change
anything here):

http://www.openstreetmap.org/copyright;
license="http://opendatacommons.org/licenses/odbl/1-0/;>
 
 
 
  
  
  
  
  
 


All three commands import the way into the database but the layer column
is empty as expected for the first and the second one.

How can your bug be reproduced?

Best regards

Michael



-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Processing dual carriageway highways into one linestring?`

2019-10-18 Thread Michael Reichert
Hi Joseph,

Am 18.10.19 um 04:29 schrieb Joseph Eisenberg:
> Has anyone implemented a rendering that handles dual-carriageway
> roads, or designed a way to import such ways into a (rendering)
> database as a single linestring midway between the two OSM ways?

We had a student at Geofabrik working on a programme to conflate
multiple parallel line strings of roads and railways as part of his
diploma thesis.

Author: Arne Johanessen
Title: Algorithmen zur automatisierten Generalisierung durch
Zusammenfassung von Linienzügen in OpenStreetMap für konkrete Spezialfälle
(translated title: Algorithms for automatic generalisation by conflation
of linestrings in OpenStreetMap for specific special cases)
Year: 2018

Full text PDF (German) is available at
http://www.geofabrik.de/media/2018-03-02-johannessen-diplomarbeit-generalisierung.pdf

The source code is available at
https://github.com/johannessen/combiner
The software is licensed under BSD 3-clause license.

Best regards

Michael

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Getting historical entries - now deleted

2019-10-11 Thread Michael Reichert
Hi Jóhannes,

Am 11.10.19 um 12:44 schrieb Jóhannes Birgir Jensson:
> I'm trying to figure out how to get historical entries of bus stops in 
> Iceland but other than getting lots of old snapshots and parsing them I'm out 
> of ideas.
> 
> Any optimal route to do this?

Take the historic planet file extract for Iceland from
https://osm-internal.download.geofabrik.de/europe/iceland.html and
iterate over the contents of the file. It is ordered by type (node, way,
relation), ID and version. If an object with ID n is visible has
highway=bus_stop in version v and not visible (i.e. deleted) in version
n+1, you have found one of the bus stops you are looking for.

There are multiple options how to do that. The following list is sorted
by performance (slowest last):

(1) Write a small C++ programme using the Osmium C++ library. You can
use
https://github.com/Nakaner/FiTaDel/blob/master/tag_change_checker.cpp as
an example. The exmaple reads a history file and counts objects which
had building=* in version v but building=yes in version v+1 if version
v+1 was uploaded by a user with a given UID.

(2) The same but in Python using Pyosmium.

(3) osmium cat --output-format=opl iceland-internal.osh.pbf \
  | grep -A 1 "highway=bus_stop" \
  | grep -v "highway=bus_stop"

Best regards

Michael



-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Karlsruhe Hack Weekend 19/20 October 2019

2019-09-16 Thread Michael Reichert
Hi,

the next Karslruhe Hack Weekend is on the 19/20 October 2019. Details:
https://wiki.openstreetmap.org/wiki/Karlsruhe_Hack_Weekend_October_2019

Best regards

Michael


-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-0
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] osm2pgsql tile expiry performance

2019-06-04 Thread Michael Reichert
Hi,

Am 29.05.19 um 21:40 schrieb j:
> WEEKLY changeset using 0.92:
> -=-=-=-
> time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \
> $PGUSER -r xml changes.osc \
> --flat-nodes /database/postgresql/OSM-FLATNODES --slim \
> --number-processes 4 --style openstreetmap-carto.style \
> --tag-transform-script openstreetmap-carto.lua -e19 -o \
> $WORKDIR_OSM/expired-tiles

Just to ensure that all important test conditions are known.

Do you mean the weekly changesets dump from
https://planet.openstreetmap.org/planet/changesets-latest.osm.bz2? Or
how did you create changes.osc or where did you download it from?
changesets-latest.osm.bz contains changeset metadata only.

Btw, "-H $PGHOST" forces connection via TCP instead of Unix sockets. If
you can use Unix sockets, -H 127.0.0.1 is not required.

Best regards

Michael

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] osm2pgsql tile expiry performance

2019-06-04 Thread Michael Reichert
Hi,

Am 30.05.19 um 00:09 schrieb j:
> I note that initial testing of this code used an extract of Europe:
> https://github.com/openstreetmap/osm2pgsql/pull/747
> 
> I did not note it earlier, but I am working with the planet. I doubt
> these issues would present themselves on a modestly sized machine
> working with Europe only.

The tests were done with Europe only which is roughly 2/3 of the whole
planet. The main difference to your test is that my test applied the
diff of a single day from download.geofabrik.de. Diffs from
download.geofabrik.de are a pure diff of the old and the new file.

I will have a deeper look into this but my test machine is currently
busy doing other work.

Best regards

Michael

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Water polygon shapefiles for Openstreetmap-carto

2019-02-26 Thread Michael Reichert
Hi,

Am 26.02.19 um 10:14 schrieb ika-chan! (OpenStreetMap):
> Hi, will this mean that the coastline will eventually no longer need to 
> follow a particular direction?

No, the coastline still has to follow a particular direction.

Many map styles (e.g. OSM Carto) do not load OSM coastline ways directly
from their rendering database but use land or sea polygons and coastline
linestrings from openstreetmapdata.com as shape files.

openstreetmapdata.com uses the OSMCoastline tool to convert the
coastline ways from OSM to land polygons, sea polygons and a curated set
of coastlines as linestrings. In order to be able to convert the
linestrings into polygons, it is required to know which side of a way is
land and which is sea because we don't map continents or oceans as
mutlipolygons.

Please note that we do not map for a specific map style only.

Best regards

Michael


-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Michael Reichert
Hi Spencer,

Am 21/12/2018 um 05.54 schrieb Spencer Gardner:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?

There are currently two serious import programmes for OSM data into
PostGIS databases – Osm2pgsql and Imposm 3. The first one is written in
C++ (formerly C), the second one in Go.

I am not aware of any well-known Python library for an import. Given the
typical use cases and the speed of existing solutions, it is unlikely
that there is any. There are Python bindings of the C++ library
Libosmium. The bindings are called Pyosmium. Libosmium is the fastest
library to deal with OSM data.

Why do you need a library for the import? Which use cases are not
covered by Osm2pgsql or Imposm 3?

Please keep in mind that OSM data is large (our planet is about 40 GB in
.osm.pbf format). Using a interpreted language is a waste of time and
energy for most purposes.

> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

Osm2pgsql runs under Windows.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Overpass API - Memory Error on 16GB RAM

2018-10-23 Thread Michael Reichert
Hi Leon,

Am 23.10.18 um 19:33 schrieb mmd:
> Am 23.10.18 um 09:17 schrieb Leon Boehmer:
>> We are trying a query which returns all highways in Europe on our
>> on-premise Overpass.
> 
> I really have no idea what your ultimate goal is, but in any case I
> would strongly recommend to download a Europe extract (e.g. from
> Geofabrik [1]), and use some filtering tool like osmium-tool [2] to
> extract highways. Store the result in whatever DB you like, could be
> even Overpass API.

osmium tags-filter -o highways-in-europe.osm.pbf europe-latest.osm.pbf
w/highway

would be the command you need.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Various types and means of account blocks

2018-09-26 Thread Michael Reichert
Hi,

Am 26.09.2018 um 17:35 schrieb Mateusz Konieczny:
> 24. Sep 2018 17:57 by frede...@remote.org :
>> Do we need a full array of permissions - "can change user name", "can
>> edit own user page", "can write personal messages", etc. - and the
>> ability to short-time suspend any and all of them?
> 
> How complicated would be implementing it? I can imagine situation where 
> otherwise
> unproblematic editor changes his username 20 times a day and blocking this
> resolves the problem - but how much effort is needed to implement this 
> compared
> to say three groups "nuke user" (for spam and troll-only accounts), "block 
> all", 
> "block mapping"?

If a user changes his username too frequently, I wonder if he has the
maturity required to edit OpenStreetMap in a productive way. Changing
the username frequently makes following the changes and analysing the
edits more difficult than necessary. Some drafts of the Organised
Editing Policy forbid frequent changes of usernames for a very good
reason (i.e. violating this rule might end in a ban).

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Various types and means of account blocks

2018-09-24 Thread Michael Reichert
Hi Frederik,

Am 24.09.2018 um 17:57 schrieb Frederik Ramm:
> Is the opposite true as well - would/should someone given a cool-off
> period for being a dick in a discussion still be allowed to do mapping?

No. Anyone who is able to edit the map data should be able and has to
answer sensible questions.

> Should a normal user block perhaps simply come in two flavours, "block
> mapping only" and "block all"?
> 
> It has been suggested that blocking *all* communication functions might
> be problematic because one thing you might expect from someone you have
> blocked is that they apologise, or set something right, which they might
> not be able to do without the ability to write messages.

If we implement the new block type proposed by ika-chan!, the DWG can
place a type 1 block (editing block) on the account and tell him to
aplogoise. If the user fails to do so, it's a sign of bad intentions and
a full block can be placed.

> Do we need a full array of permissions - "can change user name", "can
> edit own user page", "can write personal messages", etc. - and the
> ability to short-time suspend any and all of them?
> 
> Thoughts are welcome.
> 
> This also ties in somewhat with a separate discussion, on how a
> prerequisite for allowing children on the platform might be that we can
> disable the "social" functions of an account. In that case it would not
> be a short-term block, but a whole class of accounts that can edit, but
> not participate in discussions (for their own protection). I'm not sure
> that can work at all (given that the ability to contact a mapper is very
> important to us). Maybe such accounts would have to be linked to a
> "responsible" parent account who then gets the messages...

I think that any mapper should have the possibility to ask another
mapper any question on their edits. Currently, we often place a 0-hour
block on accounts who fail to respond and continue editing. What shall I
myself as a mapper do if the user whose edits look strange cannot answer
my question why he deleted the restaurant? Shall he/she restore it
because the child does not respond?

The concept of hiding minors from communications might work if they are
asked/told to map just due to their ability to draw geometries, e.g. in
an enclosed system or for mapping a few villages in an area where no
data has been before and nobody else is mapping. But once they edit
existing data, "conflicts" (questions and comments) with the community
will arise.

It might be possible to solve these issues but figuring out solutions is
up to those who need minors below 16 years mapping.

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] mod_tile/renderd serving metatiles

2018-08-25 Thread Michael Reichert
Hi,

Am 25.08.18 um 04:27 schrieb Lynn W. Deffenbaugh (Mr):
> Has anyone considered, designed, and/or implemented an extension to
> mod_tile to support serving raw metatile files to clients?
> 
> I've got my own tile server running using mod_tile and renderd and my
> own application that consumes OSM-compatible tiles.  I really don't like
> seeing the individual tiles arrive on my Android devices and would like
> to extend the knowledge of the metatile format into my client
> application.  That way, a single round-trip to the server brings back
> multiple tiles rather than one per round-trip.  The client caches tiles
> in an MBtiles database, so it can easily extract the individual tiles
> from the metatile and locally access them as needed.

Large tile server providers using mod_tile have an additional cache
(Squid or Varnish) between their tile servers and their end users.

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] GDPR implementation on planet.osm.org

2018-06-23 Thread Michael Reichert
Hi Roland,

Am 2018-06-20 um 20:16 schrieb Roland Olbricht:
> On the technical side, things are even worse. The elephant in the room
> is OAuth. OAuth is built on in particular the assumptions that
> - the consumer ("the website") acts stateful
> - sessions are relatively long-lived, i.e. some seconds to some hours
> - the identity provider has the cross-origin assets
> All three are not true for Overpass API which means that I have to work
> around OAuth or significantly mess with it.
> 
> For example, implementing to have sessions on Overpass API will require
> to develop a full-fledged security system to deal with the hundres of
> potential modes of attacks on session based systms. Even if that works,
> the median runtime for a request on Overpass API is well below a second,
> and just the roundtrip times for the OAuth threesome communication sum
> up to more. We have not even started to talk about the plethora of error
> messages that need to be formulated, explained, and implemented.

You do not need the full roundtrip for each request. I have implemented
the authentication of the protected part of Geofabrik's download service
(https://osm-internal.download.geofabrik.de/). Its source code can be
found at https://github.com/geofabrik/sendfile_osm_oauth_protector

1. If a user requests a protected resource https://HOST/PATH for the
first time, he will receive the landing page containing a link to
https://HOST/PATH?show_landing_page.

2. If he follows this link, the web application will check if he
attached a cookie to his request. If no cookie was attached, the
application will retrieve a temporary request token from
https://www.openstreetmap.org/oauth/request_token and reply with a
redirect (302 Found) to
https://www.openstreetmap.org/authorize?oauth_token=X_token_secret=Y_callback=https://HOST/PATH?oauth_token_secret_encry=Y_ENCRYTPED

3. The browser will call the URL in the Location header of the response
of item 2. If the user is already logged in into OSM, he will be asked
to grant a permission to the application "Geofabrik Downloads".
Otherwise, he has to log in first.

4. If the user grants the permission and clicks on "Grant permissions",
his browser fill send a HTTP POST request to
https://www.openstreetmap.org/oauth/authorize. The OSM website will
respond with code 302 and pointing him to
https://HOST/PATH?oauth_token_secret_encr=Y_ENCRYPTED_token=X

5. The user calls (HTTP GET)
https://HOST/PATH?oauth_token_secret_encr=Y_ENCRYPTED_token=X. The
web application of the download server recognizes the URL parameters
oauth_token and oauth_token_secret. The web application retrieves a
permanent OAuth access token from the OSM API by calling
https://www.openstreetmap.org/access_token. If that works, it is able to
call https://api.openstreetmap.org/api/0.6/user/details (with the
permanent access token in the HTTP Authentication header). If this
request does not fail, the access token is valid and the web application
has ensured that the client has a valid OSM account. The web application
sets a cookie as described in
https://github.com/geofabrik/sendfile_osm_oauth_protector/blob/master/doc/cookie.md
and responds with the requested resource.

The cookies contains the login status (unencrypted, unsigned), the name
of the key set which was used by the server to encrypt and sign the
cookie and a encrypted and signed part consisting of the access token,
the access token and the expiry date (48 hours).

6. The client sends this cookie with all future requests to the server.
The server decrypts the cookie and checks the signature. If it is ok and
the expiry date has not passed yet, the request is answered immediately
without further OAuth round trips. If the expiry date has passed, the
server responds with a redirect (code 302) as described in item 2.

Our solution does not need any session management on our side. The
session IDs are stored in the cookies. They are encrypted and signed to
prevent clients to manipulate them (or the expiry date).

It would be possible to avoid the round trip every 48 hours if the
server calls https://api.openstreetmap.org/api/0.6/user/details again
using the permanent access token (it's in the cookie). This means, you
could revalidate the validity of the OSM account every 48 hours.
However, this feature bears a security risk. It is implemented in my
tool but we at Geofabrik decided not to use it. If a user accidentally
publishes his cookie on GitHub (e.g. forgotten to remove it from the
invokation of curl), someone else could use it forever (until the access
token is revoked by the user which usually does not happen). Instead, we
require the user to re-enter his OSM account credentials every 48 hours
and require such malicious users to publish their OSM account credentials.

> On top of that, the OAuth idea means that each and every sequence of
> user data access will trigger an event on the central OSM OAuth server.
> This is quite Orwellian. Even if you do not store that 

Re: [OSM-dev] issue with geofabrik europe update

2018-05-17 Thread Michael Reichert
Hi Flo,

Am 17.05.2018 um 16:49 schrieb Florian Lohoff:
> On Wed, May 16, 2018 at 05:50:19PM +0200, Frederik Ramm wrote:
>> Hi,
>>
>> On 05/16/18 14:43, Julien Fastré wrote:
>>> We had a strange issue with a europe diff update from geofabrik: the
>>> diff file is not a valid xml.
>>
>> And that's entirely my fault for using "sed" to modify a couple of .osc
>> files around the beginning of May. Sorry for that! It was after we got
>> rid of uid/user fields in .osc files, and it turned out that some people
>> has issues with the reduced files, so we decided to put dummy uid/user
>> fields back in, and for the old files I quickly did that with a too
>> broad search-and-replace command ;)
> 
> So you actually say that geofabrik pbf/osc files dont contain valid
> usernames anymore?
> 
> I was wondering why i saw numerical usernames but if thats the case
> i can drop usernames from my QA tools.

OSM files which are available on our public server without log-in don't
contain user names, user IDs and changeset IDs. To be precise, these
fields are set to zeros/empty strings because Libosmium < 2.14 (and
Osm2pgsql < 0.96) and Osmosis (all versions) expect these fields in the
DenseInfo block of PBF files.

You can get files with full metadata at
https://osm-internal.download.geofabrik.de

Some information on the access using scripts can be found at
https://blog.geofabrik.de/?p=484
https://github.com/geofabrik/sendfile_osm_oauth_protector#documentation

I am working on cookie support on Osmosis. A preview can be found at
[1]. Someone else is working on the same for pyosmium-up-to-date.

Best regards

Michael


[1] https://github.com/openstreetmap/osmosis/pull/40

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Working with OSM data with less or no metadata

2018-02-14 Thread Michael Reichert
Hi,

people are talking about potential changes to the amount of (personal)
data distributed by OSM, in the light of new data protection laws
becoming effective in the EU this May. There haven't been any official
statements by the OSMF but discussions are going on in the LWG [1].

Even though it is still unclear what the concrete steps will be, I have
done some experiments. How well do our existing tools behave if you feed
them with OSM data that has less metadata than usual, or no metadata at
all? I have set up a test suite which tests Osmium-Tool (which uses the
Libosmium library; master branch), Osmosis 0.44.1 and Osmconvert 0.6.

The test suite is availabe at
https://github.com/geofabrik/metadata-test/
and consists of a Bash script. You need to have osmium, osmosis and
osmconvert in your path (or you have to modify the script a bit). The
test suite comes with its own hand crafted test data which will be first
converted to PBF by Osmium. Afterwards all three tools will prove
themselves in the following challenges:

- converting XML to PBF
- converting PBF to XML
- converting XML to XML
- applying a diff
- deriving changes between two OSM files

All challenges are run four times, one iteration with full metadata, one
with timestamp and version fields, one with version field only and one
without any metadata. Some PBF challenges will also have two variants –
one with DenseNodes and one without.

The results are files located in the output/ directory. You have to
inspect them manually, I have not written a tool to parse them and
output how many tests failed.

*Results*
I compiled the results into a spreadsheet. You can download it at
https://github.com/geofabrik/metadata-test/raw/master/table.ods

To sum them up:
- Osmium is the only programme which passes all format conversion tests.

- Osmosis cannot read any XML (OSM and OSC) files without timestamp and
version fields.

- Osmosis and Osmconvert [2] treat all metadata fields in the DenseInfo
message of the PBF format as mandatory. However, the format
specification doesn't declare these fields as mandatory. Therefore, they
write default values into PBF files if the input lacks these fields:
version="-1" timestamp="1969-12-31T23:59:59Z" changeset="-1" (Osmosis [3]),
timestamp="1970-01-01T00:00:01Z" changeset="1" version="1" (Osmconvert)
This partially applies to the XML output of Osmosis, too.

- Deriving a diff file of the changes between two OSM files only works
if both files have the same amount of metadata. If one file contains
less or more metadata, all objects will appear in the diff file with
their new metadata and bloat it up. The question is whether this is the
desired behaviour (i.e. the ability to clean a file from metadata using
large diffs) or if this behaviour is not desired and the tools
generating diffs should compare the tags, location and members of
objects which have the same ID but different metadata.

- Some tools have bugs which lead to wrong diffs (e.g. missing
modifications) if some metadata fields are missing.

Best regards

Michael


[1]
https://wiki.osmfoundation.org/wiki/Working_Group_Minutes#Licensing_Working_Group
[2] Osmium also had this bug. But it was fixed on the master branch a
few days ago.
[3] Osmium cannot parse negative version numbers and throws an exception.


-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Scale of downloaded images seems to vary.

2018-01-11 Thread Michael Reichert
Hi,

Am 11.01.2018 um 10:54 schrieb Tom Hughes:
> Looks like we are using 90dpi actually. The key code is here:
> 
> https://github.com/openstreetmap/chef/blob/master/cookbooks/tile/templates/default/export.erb#L121
> 
> 
> We project the bounding box from EPSG:4326 lat/lon to spherical mercator
> which gives us coordinates on a projected sheet measuring roughly
> 40075016m on each edge (2 x PI x assumed earth radius).
> 
> We then divide by the scale factor and then convert metres to pixels by
> dividing by 0.00028 which comes from:
> 
>   1 / 39.701 / 90
> 
> Where we divide by 39.701 to convert from metres to inches and then by a
> further 90 to convert to pixels at 90 dpi.

Nik4 goes one step beyond and multiplies the scale by (1 / cos(lat)).
lat is the latitude of the centre of the rendered image. That's a pretty
good approximation.
https://github.com/Zverik/Nik4/blob/master/nik4.py#L289

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev