[Taginfo-dev] started working on : Taginfo Dockerizing + Automatic configurations ( for every country )

[Imre Samu]

TLDR:
-  just you know about this   ( and avoid parallel working )
-  https://github.com/ImreSamu/osmdtaginfo  ( temporary home  ;   temporary
project name )
-  I have a *dockerized working prototype *for  all  "Central-America "
countries  ( Haiti, Cuba, .. )
-  [in a next 4-8-12-*  weeks] I hope :
   -  We have a  taginfo customization for every country   ( and a script
for a quick  :  "Region" / "City" / "District" / "HOTtask" / "POSM"  /
"Mapping Party"   setup )
   -  I hope I can send an online test link ... [  but I don't have a
financial resources for hosting for a long time ]

===
Hi,

One of my favorite tool is the osm Taginfo, and my dream:  creating a
Taginfo instance for every country in the Earth.
And on the "State of the Map 2016"  -  Taginfo -  Birds of Feather meeting
 - we discussed something about this  ~
so I have started working on this topic.
- Current *_temporary_ * repository :   https://github.com/ImreSamu/os
mdtaginfo
- The license is the same  ~  Taginfo  (GPL 3.0)
- And I hope later I can move to this  ->  to the  taginfo repo  [
https://github.com/taginfo ]

Status:

*- The automatic config file / map / generations is working* ( basic level
, and need more test and optimization )
   and for* Central-America*  I uploaded to the repository.
   for example Haiti : https://github.com/ImreSamu/
osmdtaginfo/tree/master/taginfo_data/ht

- I* have created 2 Docker Images  *( https://github.com/ImreSamu
/osmdtaginfo/tree/master/docker )
  -  *taginfo_view*  ( for only for view - Alpine based,configured for
: max 60Mb RAM )
  - * taginfo_job*( for taginfo setup & taginfo data processing jobs &
development  )

so for the every country  ( in theory )  a 32 Gb RAM is enough
-  *taginfo_view docker : *:  RAM =   (  220 country x 60Mb  )   ~  14 - 20
Gb RAM
-  serial batch processing *taginfo_job ( 1 x 4Gb RAM )   *

T*he reverse proxy settings and a lot of other things is missing ...*

My vision ( but the links not exist yet )
   

https://cr.taginfo.openstreetmap.org // taginfo for : "Costa Rica"
https://cu.taginfo.openstreetmap.org // taginfo for : "Cuba"
https://dm.taginfo.openstreetmap.org // taginfo for : "Dominica"
https://do.taginfo.openstreetmap.org // taginfo for : "Dominican Republic"
https://gd.taginfo.openstreetmap.org // taginfo for : "Grenada"
https://hn.taginfo.openstreetmap.org // taginfo for : "Honduras"
https://ht.taginfo.openstreetmap.org // taginfo for : "Haiti"

   


 status:  *this is only a Proof of Concept * ==

if you have a (latest)   *docker* and *docker-compose* installed,  you can
test yourself.

*git clone https://github.com/ImreSamu/osmdtaginfo.git
*
*cd osmdtaginfo*
*./test.sh* ( download Geofabrik
central-america-latest.osm.pbf ~ 246 Mb ,  Build 2 docker containers,  +
Minimal tests ..)

and

*make test_ht  * (   test for Haiti)


check/adapt  the container ip  , ( docker inspect )  and open with the
browser :   ~ http://172.21.0.2:4567/


*Available (all) Central-America configurations: *

[ if you are testing multiple countries, be aware *the browser caching*
( customized background image, and logo sometimes not refreshed ..  I will
change this in the next version.
 on: Chrome: *Hold down Ctrl and click the Reload button! *   ]


make test_ag # test taginfo for: "Antigua and Barbuda"
make test_ai # test taginfo for: "Anguilla"
make test_bb # test taginfo for: "Barbados"
make test_bs # test taginfo for: "The Bahamas"
make test_cr # test taginfo for: "Costa Rica"
make test_cu # test taginfo for: "Cuba"
make test_dm # test taginfo for: "Dominica"
make test_do # test taginfo for: "Dominican Republic"
make test_gd # test taginfo for: "Grenada"
make test_hn # test taginfo for: "Honduras"
*make test_ht # test taginfo for: "Haiti"*
make test_jm # test taginfo for: "Jamaica"
make test_kn # test taginfo for: "Saint Kitts and Nevis"
make test_ky # test taginfo for: "Cayman Islands"
make test_lc # test taginfo for: "Saint Lucia"
make test_ms # test taginfo for: "Montserrat"
make test_ni # test taginfo for: "Nicaragua"
make test_nl # test taginfo for: "The Netherlands"
make test_pa # test taginfo for: "Panama"
make test_sv # test taginfo for: "El Salvador"
make test_tc # test taginfo for: "Turks and Caicos Islands"
make test_tt # test taginfo for: "Trinidad and Tobago"
make test_vc # test taginfo for: "Saint Vincent and the Grenadines"
make test_vg # test taginfo for: "British Virgin Islands"


Ps.
- probably  the 2  letter country code is not enough ...  ( for example -
there is a "The Netherlands" in the Central America  and in the Europe,
 and it is 'bad' for the geographical distribution maps ..
 so later probably I will add a 'continent' prefix or postfix
 like
 - ca - central-america ( ca-ht =Haiti   , ca-nl  = The Nederlands in the
Central 

[Taginfo-dev] 1week test: taginfo-dev.opengeodata.hu - 622 areas - every country and some new experimental features

[Imre Samu]

This is a Proof of Concept of my vision -  about customizing taginfo for
countries, regions ...

dev site: http://taginfo-dev.opengeodata.hu
1 week test:   shutdown time:  *~ 2018-aug-20*


Main changes:

*-  620 areas  - not refreshing *
  ( = 620 docker services running in a simple cloud machine.
32Gb RAM,   slow CPU :  Intel(R) Atom(TM) CPU  C2750  @
2.40GHz,   8 core ,  ~ 600Gb Disk )

*-  2 new experimental reports*:

  "QA-Normalized name differences (Experimental)"
example:
http://eu-at.taginfo-dev.opengeodata.hu/reports/normalized_names
( I hope - this will be useful for the localized
https://github.com/osmlab/name-suggestion-index
 ( see
https://github.com/osmlab/name-suggestion-index/issues/11 )
The result can be download as an xlsx file:
http://eu-at.taginfo-dev.opengeodata.hu/download/normalized_names.xlsx

  "QA-Problematic tags (Experimental)" [ still a lot of bugs,   for
example:  checking access type of tags  is not perfect yet, sorry ]
 example:
http://eu-at.taginfo-dev.opengeodata.hu/reports/problematic_tags
 .xlsx result:
http://eu-at.taginfo-dev.opengeodata.hu/download/problematic_tags.xlsx

*- `name` support for tags  ( Experimental )  *

 example:
http://eu-at.taginfo-dev.opengeodata.hu/tags/amenity=fast_food#tagnames_lang1

 or Spain   amenity=place_of_worship
 name   =
http://eu-es.taginfo-dev.opengeodata.hu/tags/amenity=place_of_worship#tagnames_lang1
 name:es  =
http://eu-es.taginfo-dev.opengeodata.hu/tags/amenity=place_of_worship#tagnames_lang2
 name:eu  =
http://eu-es.taginfo-dev.opengeodata.hu/tags/amenity=place_of_worship#tagnames_lang3
 name:ca  =
http://eu-es.taginfo-dev.opengeodata.hu/tags/amenity=place_of_worship#tagnames_lang4
 name:gl   =
http://eu-es.taginfo-dev.opengeodata.hu/tags/amenity=place_of_worship#tagnames_lang5

   or Switzerland   amenity=bank
 name   =
http://eu-ch.taginfo-dev.opengeodata.hu/tags/amenity=bank#tagnames_lang1
 name:en  =
http://eu-ch.taginfo-dev.opengeodata.hu/tags/amenity=bank#tagnames_lang2
 name:de  =
http://eu-ch.taginfo-dev.opengeodata.hu/tags/amenity=bank#tagnames_lang3
 name:fr=
http://eu-ch.taginfo-dev.opengeodata.hu/tags/amenity=bank#tagnames_lang4
 name:it=
http://eu-ch.taginfo-dev.opengeodata.hu/tags/amenity=bank#tagnames_lang5

-
source code - for the dockerization:
https://github.com/taginfo/dockerized-taginfo
taginfo changes:
https://github.com/taginfo/taginfo/compare/master...ImreSamu:name_tabs_v2

This is an experimental - and not official changes,   probably some
changes will be converted as a "taginfo plug-in".
The 2 experimental reports implemented in Julia language (
https://julialang.org/ )

-

Regards,
   Imre
___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev

[Taginfo-dev] dockerized-taginfo - for every country

[Imre Samu]

Hi,

If you have some time,
please help testing my development taginfo service.


3 test continent:
==
http://taginfo-dev.opengeodata.hu/africa/  ~ 65 Taginfo
service
http://taginfo-dev.opengeodata.hu/antarctica/   ~  1 Taginfo service
http://taginfo-dev.opengeodata.hu/central-america/  ~ 33 Taginfo service

= 99 Test taginfo service ( Dockerized - on the same server )

for example:
* in Antartica   http://aq-aq.taginfo-dev.opengeodata.hu/keys/amenity#values
We have
- 5 toilets
- 2 Bank
- 1 Atm
- 2 cafe
( The Overpass link from Taginfo is not working for Antartica:
  https://github.com/drolbr/Overpass-API/issues/421 )

or other countries:
* South Africa  :  http://af-za.taginfo-dev.opengeodata.hu/
* Costa Rica : http://ca-cr.taginfo-dev.opengeodata.hu/
* Morocco : http://af-ma.taginfo-dev.opengeodata.hu/
* ...

Github: https://github.com/taginfo/dockerized-taginfo

Regards,
 Imre
___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev

Re: [Taginfo-dev] dockerized-taginfo - for every country

[Imre Samu]

I have added some new areas ( no daily refresh )

Asia:http://taginfo-dev.opengeodata.hu/asia/   (  5
area:  Indonesia, Sri Lanka, Myanmar, Mongolia, Nepal )
Australia-Oceania:  http://taginfo-dev.opengeodata.hu/australia-oceania/  (
8 area )
Europe :  http://taginfo-dev.opengeodata.hu/europe/  ( 5
area :   Romania, Berlin, Iceland, Istanbul )
North America http://taginfo-dev.opengeodata.hu/north-america/   (
British Columbia, Greenland, Mexico, California, Mississippi )
South Americahttp://taginfo-dev.opengeodata.hu/south-america/  (
Bolivia, Columbia, Peru, Paraguay )
Russia:http://taginfo-dev.opengeodata.hu/russia/ (
Altai Rep, Komi Rep, Saint Petersburg, Republic of Mordovia, Tatarstan )

if somebody wants to test some special area -please write me!


My biggest problem:
-  Antimeridian  -  example:   New Zealand Maps  not working (
http://ao-nz.taginfo-dev.opengeodata.hu/keys/highway#map )
Planned Quick solution:I will  modify the New Zealand  ( and
other countries ) Polygons   - so it will not touch the  Antimeridian(
but the statistics will be worse )

And there are a lot of small problems ..
-  small area background map is not perfect ( now I am using natural earth
raster data,  )
-  job processing is not perfect, detecting errors is not always easy.

Current Status:
- 131 running taginfo container ( alpine linux + ruby )  + proxy +
webserver ...   (  my dev system = 15Gb RAM , 6 core )

now the memory:
root@scw-343bdf:/mnt/data/dockerized-taginfo# free -m
  totalusedfree  shared  buff/cache
 available
Mem:  150192990 974 174   11054
 11535
Swap: 0   0   0

For refreshing the data ( taginfo processing)  at least 5-10 GB Free RAM
need ...  but for the big countries, probably more ...  ( Canada, US,
France, Germany )

For detecting "Long tail" errors I am using the high  detail settings,
*"min_count_tags": 1,*
// Tag combinations not appearing this often are not written to
database.
*"min_tag_combination_count": 1,*
// Minimum number of tag occurances for creating a map
*"min_count_for_map": 1,*
// Minimum number of relations per type to make this
// relation type "interesting", ie. to make it show
// up as a relation type.
*"min_count_relations_per_type": 1,*

With some "basic"   source code customisation  ( some is not optimal yet )

 && sed -i 's/100/10/g'
 /osm/taginfo/sources/master/master.sql \
 && sed -i 's/(default 1)/(default 10)/g'
 /osm/taginfo/web/lib/api/v4/keys.rb \
 && sed -i 's/min_count = 1/min_count = 10/g'
 /osm/taginfo/web/lib/api/v4/keys.rb \
 && sed -i 's/count_all_common >= 1/count_all_common >= 10/g'
/osm/taginfo/web/lib/api/v4/keys.rb \



For the future I have 2 plan:
A: Finding a sponsor and running 200-300  important  areas in a container

B: Creating about ~4000 config files ( in the github repo ) and with a
simple quickstart everybody can run the selected local taginfo.

the planned process for the "B"   ( not working yet )

git clone https://github.com/taginfo/dockerized-taginfo.git
cd dockerized-taginfo
make build
cd ./service/eu// select Europe
make taginfo-eu-de-be //  Create "Berlin" taginfo with downloading and
processing osm data, and start the web frontend

and open the browser: 127.0.0.1:20132


Regards,
 Imre





2018-03-13 23:17 GMT+01:00 Imre Samu <pella.s...@gmail.com>:

> Hi,
>
> If you have some time,
> please help testing my development taginfo service.
>
>
> 3 test continent:
> ==
> http://taginfo-dev.opengeodata.hu/africa/  ~ 65 Taginfo
> service
> http://taginfo-dev.opengeodata.hu/antarctica/   ~  1 Taginfo
> service
> http://taginfo-dev.opengeodata.hu/central-america/  ~ 33 Taginfo service
> 
> = 99 Test taginfo service ( Dockerized - on the same server )
>
> for example:
> * in Antartica   http://aq-aq.taginfo-dev.opengeodata.hu/keys/amenity#
> values
> We have
> - 5 toilets
> - 2 Bank
> - 1 Atm
> - 2 cafe
> ( The Overpass link from Taginfo is not working for Antartica:
>   https://github.com/drolbr/Overpass-API/issues/421 )
>
> or other countries:
> * South Africa  :  http://af-za.taginfo-dev.opengeodata.hu/
> * Costa Rica : http://ca-cr.taginfo-dev.opengeodata.hu/
> * Morocco : http://af-ma.taginfo-dev.opengeodata.hu/
> * ...
>
> Github: https://github.com/taginfo/dockerized-taginfo
>
> Regards,
>  Imre
>
___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev

Re: [Taginfo-dev] Getting matching maps on Taginfo gb

[Imre Samu]

Hi Shaun,

sorry, now I can't debug your values - but as I remember it is not easy.

I am calculating with this ugly SQL function - for every country.
*
https://github.com/taginfo/dockerized-taginfo/blob/master/setup/setup_xtaginfo_admin.sql#L42

and my result for GB  - this.
https://github.com/ImreSamu/dockerized-taginfo/blob/sotm2018/service/eu/gb/taginfo-config.json
 - but this is smaller than yours
( ~ 2-3 day online: http://207.180.196.156:20142/keys/highway#map )
 included : https://en.wikipedia.org/wiki/Rockall - so it is strange ..

---
and calling with your coordinate - my result.
===
osm=# select get_xtaginfo('gb', -12.35,2.98, 49.47, 61.2) ;
NOTICE:  OK   R# = gb, 18 , 279.000 , 216.000 ,0.170 ,0.270
get_xtaginfo
-
 {279,216,-12.35,3.150,49.47,61.470}
(1 row)
==
so the geneated image:  279 x 216  pixel
and the coordinates=  -12.35,3.150,49.47,61.470
but  I can't test ..

Imre






Shaun McDonald  ezt írta (időpont: 2019. ápr.
20., Szo, 11:20):

> Hi,
>
>
> I’m trying to get the background map to match the Taginfo generated map
> for the GB Taginfo. I seem to be missing something from the instructions on
> the OSM wiki, as the background maps. With an earlier version I was getting
> too much included vertically.
>
>
> https://wiki.openstreetmap.org/wiki/Taginfo/Installation#Adapting_the_map_view
>
> My working is in
> https://docs.google.com/spreadsheets/d/1rnO5jqZAirsFankimavetobulNE2f5MJi4Z0PA8h0Dc/edit?usp=sharing
>
> I’ve followed
> https://switch2osm.org/manually-building-a-tile-server-18-04-lts/ to
> produce a map DB.
>
> Map image script:
> cat create_taginfo_map.py
> import mapnik
>
> m=mapnik.Map(330,345)
> mapnik.load_map(m, 'openstreetmap-carto/taginfo_outline.xml')
>
> bbox = mapnik.Box2d(-9,49.8,2.5,60.8)
> m.zoom_to_box(bbox)
>
> im = mapnik.Image(330,345)
> mapnik.render(m,im)
> im.save('gb_map.png', 'png256’)
>
> The highway map is a good example showing the issue:
> https://taginfo.openstreetmap.org.uk/keys/highway#map
>
> head openstreetmap-carto/taginfo_outline.xml
> 
> 
> 
>  
>1
>2
>
>
> name="bounds">-180,-85.05112877980659,180,85.05112877980659
>0,0,4
>
>
> Taginfo config segment:
>// For the geodistribution map. See the wiki documentation about these
> settings.
>"geodistribution": {
>"left": -9,
>"bottom": 49.8,
>"right": 2.5,
>"top": 60.8,
>"width": 345,
>"height": 330,
>"scale_image": 1,
>"scale_compare_image" : 1,
>//"background_image": "/img/mapbg/world.png",
>"background_image": "/img/mapbg/gb_map.png",
>// optional attribution for base image, will show up below all maps
>"image_attribution": ""
>},
>
> I can’t quite figure out what I’m doing wrong to get the maps to line up
> correctly while following the instructions on the wiki. I’m so close now.
> Would anyone be able to provide some pointers on how to fix?
>
> Shaun
>
>
> ___
> Taginfo-dev mailing list
> Taginfo-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/taginfo-dev
>
___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev

Re: [Taginfo-dev] Getting matching maps on Taginfo gb

[Imre Samu]

>Thank you, these figures have done the trick.

good news :)
( I see : https://taginfo.openstreetmap.org.uk/taginfo/config )

some small comments:
#1.
  you can add a configturbo.wizard_area =  "in 'United Kingdom'"

https://github.com/ImreSamu/dockerized-taginfo/blob/sotm2018/service/eu/gb/taginfo-config.json#L71
   but this is containing Northern Ireland
  ( I have checked the  "Great Britain" relation, but this is smaller
https://www.openstreetmap.org/relation/6038068 , a lot of islands not
included )

#2.
with high precision extracting - you can remove the extra tags in Ireland
( ~ south coastline )  or in France
* https://taginfo.openstreetmap.org.uk/keys/natural#map  vs.
http://207.180.196.156:20142/keys/natural#map
I am running an extra "osmium extract --strategy simple" on the input
data:
https://github.com/taginfo/dockerized-taginfo/blob/master/sh/osm_split.sh#L47

Imre





Shaun McDonald  ezt írta (időpont: 2019. ápr.
22., H, 0:27):

> Hi Imre,
>
> Thank you, these figures have done the trick.
>
> The coordinates are in a different order to which Taginfo uses, however
> swapping the middle 2 numbers produces a good map.
>
> Shaun
>
> On 20 Apr 2019, at 17:04, Imre Samu  wrote:
>
> Hi Shaun,
>
> sorry, now I can't debug your values - but as I remember it is not easy.
>
> I am calculating with this ugly SQL function - for every country.
> *
> https://github.com/taginfo/dockerized-taginfo/blob/master/setup/setup_xtaginfo_admin.sql#L42
>
> and my result for GB  - this.
>
> https://github.com/ImreSamu/dockerized-taginfo/blob/sotm2018/service/eu/gb/taginfo-config.json
>  - but this is smaller than yours
> ( ~ 2-3 day online: http://207.180.196.156:20142/keys/highway#map )
>  included : https://en.wikipedia.org/wiki/Rockall - so it is strange ..
>
> ---
> and calling with your coordinate - my result.
> ===
> osm=# select get_xtaginfo('gb', -12.35,2.98, 49.47, 61.2) ;
> NOTICE:  OK   R# = gb, 18 , 279.000 , 216.000 ,0.170 ,0.270
> get_xtaginfo
> -
>  {279,216,-12.35,3.150,49.47,61.470}
> (1 row)
> ==
> so the geneated image:  279 x 216  pixel
> and the coordinates=  -12.35,3.150,49.47,61.470
> but  I can't test ..
>
> Imre
>
>
>
>
>
>
> Shaun McDonald  ezt írta (időpont: 2019. ápr.
> 20., Szo, 11:20):
>
>> Hi,
>>
>>
>> I’m trying to get the background map to match the Taginfo generated map
>> for the GB Taginfo. I seem to be missing something from the instructions on
>> the OSM wiki, as the background maps. With an earlier version I was getting
>> too much included vertically.
>>
>>
>> https://wiki.openstreetmap.org/wiki/Taginfo/Installation#Adapting_the_map_view
>>
>> My working is in
>> https://docs.google.com/spreadsheets/d/1rnO5jqZAirsFankimavetobulNE2f5MJi4Z0PA8h0Dc/edit?usp=sharing
>>
>> I’ve followed
>> https://switch2osm.org/manually-building-a-tile-server-18-04-lts/ to
>> produce a map DB.
>>
>> Map image script:
>> cat create_taginfo_map.py
>> import mapnik
>>
>> m=mapnik.Map(330,345)
>> mapnik.load_map(m, 'openstreetmap-carto/taginfo_outline.xml')
>>
>> bbox = mapnik.Box2d(-9,49.8,2.5,60.8)
>> m.zoom_to_box(bbox)
>>
>> im = mapnik.Image(330,345)
>> mapnik.render(m,im)
>> im.save('gb_map.png', 'png256’)
>>
>> The highway map is a good example showing the issue:
>> https://taginfo.openstreetmap.org.uk/keys/highway#map
>>
>> head openstreetmap-carto/taginfo_outline.xml
>> 
>> 
>> 
>>  
>>1
>>2
>>
>>
>>> name="bounds">-180,-85.05112877980659,180,85.05112877980659
>>0,0,4
>>
>>
>> Taginfo config segment:
>>// For the geodistribution map. See the wiki documentation about these
>> settings.
>>"geodistribution": {
>>"left": -9,
>>"bottom": 49.8,
>>"right": 2.5,
>>"top": 60.8,
>>"width": 345,
>>"height": 330,
>>"scale_image": 1,
>>"scale_compare_image" : 1,
>>//"background_image": "/img/mapbg/world.png",
>>"background_image": "/img/mapbg/gb_map.png",
>>// optional attribution for base image, will show up below all maps
>>"image_attribution": ""
>>},
>>
>> I can’t quite figure out what I’m doing wrong to get the maps to line up
>> correctly while following the instructions on the wiki. I’m so close now.
>> Would anyone be able to provide some pointers on how to fix?
>>
>> Shaun
>>
>>
>> ___
>> Taginfo-dev mailing list
>> Taginfo-dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/taginfo-dev
>>
>
>
___
Taginfo-dev mailing list
Taginfo-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/taginfo-dev