> and store the information in the "central metadata repository"(?) .. ( with a good license )
Yes, you're right. We need a group of people working on a specification and on tools to make it easier to validate and to use data for both mappers and developers. But we need people with knowleadge. Maybe we can create a seperate mailing list?
Cheers
Sören Reinecke alias Valor Naram
-------- Original Message --------
Subject: Re: [OSM-dev] tagtransform for OSM - An effort make tagging and using OSM data easier; bridging different worlds together
From: Imre Samu
To: Sören Reinecke
CC: OSM-Dev Openstreetmap
> There are some available options I see: ....
> What do you prefer or do you have another options to addIMHO:it is a hard problem .. ( we need more use cases ... for find the global optimum )on the other hand - I am interested in your Proof of Concept solutions.Probably we need a "Metadata Working Group" to collect this type of informations,and store the information in the "central metadata repository"(?) .. ( with a good license )related - with this discussions: https://www.openstreetmap.org/user/SomeoneElse/diary/391484Sharing some use cases :iD Editor - tag-transformation metadataThe iD editor has a lot of metadata , probably we can analyze ..the tag transformation metadata is very simple, and it has ~302 transformation rules.Data license : ISCexample:{ "old": {"aerialway": "canopy"},
"replace": {"aerialway": "zip_line"}
},
{ "old": {"aeroway": "aerobridge"},
"replace": {"aeroway": "jet_bridge", "highway": "corridor"}
},
{ "old": {"access": "public"},
"replace": {"access": "yes"}
},or a little more complex:{ "old": {"building:type": "*"},
"replace": {"building": "$1"}
},Deprecated tags OSM wiki : https://wiki.openstreetmap.org/wiki/Deprecated_featuresiD Editor - Discarded tagssee other discardable tags (JOSM,Potlach, ) https://wiki.openstreetmap.org/wiki/Discardable_tagsImposm3 has a special tag transformation typestags exclude: exclude: [created_by, source, "tiger:*"]and has some special "data mapping"[1] rules:- bool / direction / enumerate /categorize / ....and I would like to add the"COALESCE()" transformations to the transformation rules..example:"name" = coalesce("name","name:en","name:de","name:fr")giggls/mapnik-german-l10n repo has a streetname abbreviation rulesexample:abbrev=regexp_replace(abbrev,'(?<!^([0-9]+([èe]?r)?e )?)Avenue\M','Ave.');
abbrev=regexp_replace(abbrev,'(?!^)Boulevard\M','Blvd.');
abbrev=regexp_replace(abbrev,'Crescent\M','Cres.');
abbrev=regexp_replace(abbrev,'Court\M','Ct');
abbrev=regexp_replace(abbrev,'Drive\M','Dr.');
abbrev=regexp_replace(abbrev,'Lane\M','Ln.');QA ToolsAnd we can collaborate with QA tools .. for validating "values"like ... URL type values .. If not valid RFC URL .. we can safely drop .."contact:webcam"
"contact:website"
"facebook"
"url"
"website:mobile"
"website:stock"
"website"other QA tools: https://wiki.openstreetmap.org/wiki/Quality_assuranceon the other hand - the validation is hard)OSRM ( routing) LUA transformationsPostgreSQLSometimes the tag transformation is in SQLfor example - the oneway is not so simple:CASE
WHEN oneway IN ('yes', '-1') THEN oneway
WHEN junction IN ('roundabout') AND (oneway IS NULL OR NOT oneway IN ('no', 'reversible')) THEN 'yes'
ELSE NULL
END AS oneway,or size of the telescope:CASE
WHEN man_made IN ('telescope') THEN
CASE
WHEN tags->'telescope:diameter' ~ '^-?\d{1,4}(\.\d+)?$' THEN (tags->'telescope:diameter')::NUMERIC
ELSE NULL
END
ELSE NULL
END AS "telescope:diameter",if we can move this type of transformation to the osm2pgsql side .. the rendering will be fasterWe need some Unicode cleaning rulesLike my favorite problems with different apostrophes :There are some "trap" tags .. detecting bad imports ( latitude, longitude, lat, lon )We can easily drop this tags .. or create a QA list for the local communityAnd there are some problematic keys ( https://taginfo.openstreetmap.org/reports/characters_in_keys )Other tag transformation problems :- converting to metrics ( "50 mph" -> km/h ; knots .. ) ( https://wiki.openstreetmap.org/wiki/Key:maxspeed#Parser )- decoding special speed codes "FR:walk"/"DE:urban" .. (https://wiki.openstreetmap.org/wiki/OSM_tags_for_routing/Maxspeed )- injecting 3rd party data ( for example Wikidata labels , driving side , country admin code )- validating 3rd party links ( wikipedia links , wikidata id-s; website urls , facebook links )- validating osm "values" from the lookup tables / regexp ( example: "building:color" )- removing nested relations ( role=subarea) https://github.com/osmcode/osmium-tool/issues/169For an ad-hoc transformation ( osm.pbf -> osm.pbf ) we can use- osm xml formatosmium cat input.osm.pbf -f xml --no-progress -o - | sed ... | osmium cat - --input-format xml -o output.osm.pbf- https://osmcode.org/pyosmium/ ( python )- https://github.com/osmcode/node-osmium ( _javascript_ )- ( osm.pbf ) Golang libs ...- ....Regards,ImreSören Reinecke <tilmanreine...@yahoo.de> ezt írta (időpont: 2019. dec. 6., P, 13:33):There are some available options I see:
a) Not working on this further.
b) Using `tagtransform for OSM` to create an own transformation specification.
c) Writing converters which convert from a format to the format of `tagtransform for OSM` and writing converters to convert from the format of `tagtransform for OSM` to another format programs can work with. Using my specification which needs to be extended to create conpactibility among different formats while ensuring that my spec can be used on its own.
What do you prefer or do you have another options to add?
Cheers
Sören Reinecke alias Valor Naram
-------- Original Message --------
Subject: Re: [OSM-dev] tagtransform for OSM - A effort make tagging and using OSM data easier; bridging different worlds together
From: Imre Samu
To: Sören Reinecke
CC: OSM-Dev Openstreetmap> I currently write a specification for tranforming tags in OpenStreetMap to make life of data customers easier.imho: we can import some good ideas from https://wiki.openstreetmap.org/wiki/Osmosis/TagTransform schema .."The tag transform Osmosis plugin allows arbitrary tag transforms to be applied to OSM data as a preprocessing step before using other tools. This allows other tools to concentrate on doing what ever they do, without having to handle numerous different tagging schemes and error corrections."imho: regexp is useful.probably the "lua" is good glue/meta language - for writing "business rules".some examples:Valhalla (routing) admin.lua ( https://github.com/valhalla/valhalla/blob/master/lua/admin.lua )Valhalla (routing) graph.lua ( https://github.com/valhalla/valhalla/blob/master/lua/graph.lua )osm2gsql - openstreetmap-carto.luaRegards,ImreSören Reinecke via dev <dev@openstreetmap.org> ezt írta (időpont: 2019. dec. 5., Cs, 15:59):_______________________________________________Hey all,I currently write a specification for tranforming tags in OpenStreetMap to make life of data customers easier. Different tagging schemes have emerged since the existence of OpenStreetMap, same are existing in parallel and a newer one deprecated an old one. Data customers without knowing the OSM community much get lost. This project aims to help developers who want to take advantage of the OpenStreetMap great database which is by the way a brilliant project. This project can also help to make tagging in OSM more orthogonal and more hassle free.I saw conflicting interests between OSM community, OSM developers like the iD developers and data customers. A renderer might need data in another way as the community contributes. The community might need another tagging scheme than a renderer. I thought how we can resolve this, how we can get all sites on "one table" and that is the idea I had come up with:A more readable version can be found here: https://github.com/ValorNaram/transformation-table-osmtags/blob/master/README.md and the principles can be found at https://github.com/ValorNaram/transformation-table-osmtags/blob/master/principles.mdExample 1: They want to have the phone number of a POI. There are some problems with this:1. They need to know both contact:phone and phone to get them all.2. They need to support them both.3. They need to remove one in case both keys are mapped on one POI. This really happens, see http://overpass-turbo.eu/s/OI2.Example 2: They want to know how many POI's have changing tables (general: facilities for changing a nappy of a baby). There are some problems with this too:1. They need to know both changing_table and the deprecated diaper to get them all.2. They need to support them both. Difficult because they're highly different tagging schemes.3. They need to remove one in case both keys are mapped on one POI. This really happens, see http://overpass-turbo.eu/s/OI5.Example 3: They want to develop a mapping tool and want to correct wrong typed in tags. There are some problems with that:1. They need to know all possible tagging schemes existing for one purpose (e.g. mapping facilities for changing the nappy of a baby).2. They need to know the right/approved/more logical scheme.3. They need to know how to convert:diaper=yesdiaper:female=yesbecomes after the transformation:changing_table=yeschanging_table:location=female_toiletBasically it bridges the OSM community and developers together, it might even resolve the conflict between iD developers and the community.The project bridges different worlds and is therefore a bridge. As bridge this project should not just connect different worlds together and by ensuring peace between those but also support exchange between those to develop a social economy of "send and receive" This project should support the "come together" of (OSM) developers and mappers.I want to hear your opinions on this and also feel not to shame to ask questions. My answers will be part of the clarification of the project "tagtransform for OSM".CheersSören Reinecke alias ValorNaram
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
_______________________________________________ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev