Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

Mateusz, rendering still happens on the server right before being sent to
the client, but it COULD be rendered on the client too, because the
vector.pbf tiles could be accessed directly, bypassing the server side
rendering.

On Sat, May 5, 2018 at 3:01 PM Mateusz Konieczny 
wrote:

>
> Thanks, I forgot/missed that rendering happens on
> client computer and vector tiles with all languages are cached.
>
> I also now have one more project on my list - look how
> vector tiles served by Wikipedia servers may be used
> (usage limits and what kind of data is provided).
>
> 4. May 2018 10:14 by yuriastrak...@gmail.com:
>
> Btw, this is already possible - Wikipedia servers let you access both
> images and raw vector tiles, so if someone wants to do the client part, it
> shouldn't be too hard.
>
> On Fri, May 4, 2018, 11:11 Yuri Astrakhan  wrote:
>
>> In reality, it is not impossible, or even that hard. If the vector tile
>> is sent to the client, than the client can decide which language to render
>> based on user preference.  The exact same code (it's all in JavaScript) can
>> be used to decide the labeling. Caching would only improve, because instead
>> of caching multiple raster tiles, it would only cache a single vector tile.
>>
>>  Mapbox.gl or open layers can both do this fairly efficiently.
>>
>>
>>
>> On Fri, May 4, 2018, 10:54 Mateusz Konieczny 
>> wrote:
>>
>>>
>>>
>>>
>>> 4. May 2018 07:36 by md...@xs4all.nl:
>>>
>>> On 2018-05-04 00:33, Joe Matazzoni wrote:
>>>
>>> No fallback is currently defined for Polish. We’ll be happy to
>>> change that if you can show community consensus.
>>>
>>>
>>> Community consensus? You mean a bunch of people who decide for the whole
>>> country (many of them have no idea of the mechanisms behind it) what the
>>> strategy is?
>>> I'm sorry to say, but that can not be consensus. This needs to be able
>>> to be configured at user level.
>>>
>>>
>>> Server resources are not infinite, separate map cache for every user
>>> would be probably unfeasible.
>>> ___
>>> talk mailing list
>>> talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk
>>>
>>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Mateusz Konieczny]

Thanks, I forgot/missed that rendering happens on 
client computer and vector tiles with all languages are cached.

I also now have one more project on my list - look how 
vector tiles served by Wikipedia servers may be used 
(usage limits and what kind of data is provided).

4. May 2018 10:14 by yuriastrak...@gmail.com :


> Btw, this is already possible - Wikipedia servers let you access both images 
> and raw vector tiles, so if someone wants to do the client part, it shouldn't 
> be too hard.
>
> On Fri, May 4, 2018, 11:11 Yuri Astrakhan <> yuriastrak...@gmail.com 
> > > wrote:
>
>> In reality, it is not impossible, or even that hard. If the vector tile is 
>> sent to the client, than the client can decide which language to render 
>> based on user preference.  The exact same code (it's all in JavaScript) can 
>> be used to decide the labeling. Caching would only improve, because instead 
>> of caching multiple raster tiles, it would only cache a single vector tile. 
>>  Mapbox.gl or open layers can both do this fairly efficiently.
>>
>>
>> On Fri, May 4, 2018, 10:54 Mateusz Konieczny <>> matkoni...@tutanota.com 
>> >> > wrote:
>>
>>>   
>>>
>>>
>>> 4. May 2018 07:36 by >>> md...@xs4all.nl >>
>>>

 On 2018-05-04 00:33, Joe Matazzoni wrote:
> No fallback is currently defined for Polish. We’ll be happy to
> change that if you can show community consensus.

 Community consensus? You mean a bunch of people who decide for the whole 
 country (many of them have no idea of the mechanisms behind it) what the 
 strategy is?
 I'm sorry to say, but that can not be consensus. This needs to be able to 
 be configured at user level.


>>>
>>>
>>>
>>>
>>> Server resources are not infinite, separate map cache for every user would 
>>> be probably unfeasible.
>>>
>>>   >>> ___
>>> talk mailing list
>>> talk@openstreetmap.org 
>>> https://lists.openstreetmap.org/listinfo/talk 
>>> 
>>>___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Mark Wagner]

Japanese has three *alphabets* (well, two syllabaries and a set of
logographs), but only one writing system.  For any given thing you want
to write, there's only one correct set of symbols to use.

-- 
Mark


On Fri, 4 May 2018 08:11:14 +0200
Jo  wrote:

> Japanese has 3 writing systems.
> 
> Jo
> 
> 2018-05-04 1:10 GMT+02:00 Daniel Koć :
> 
> > W dniu 04.05.2018 o 00:33, Joe Matazzoni pisze:
> >
> > Here is the list of languages Wikimedia supports. https://en.
> > wikipedia.org/wiki/Special:SiteMatrix
> > 
> >  .
> > As to the toolchain, if you ask that question on the project page,
> > some of our engineers will be able to respond.  Be sure to explain
> > why you’re asking, so we can answer you fully.
> >
> >
> > It's not exactly what interests me:
> >
> > a) do you want to support all these languages (not how many of them
> > are there)?
> >
> > b) how many server resources do you need for rendering 
> > languages that you want to support (not the software stack used)?
> >
> > Do you know it or should I ask the engineers anyway?
> >
> > That is a very interesting edge case we hadn’t thought of.  We’ll
> > have to look into that one; our lead engineer just wrote a ticket
> > to investigate: https://phabricator.wikimedia.org/T193815
> > .
> >  I know about Serbian. What other languages do this?
> >
> >
> > Belorusian has two writing systems (see http://openstreetmap.by for
> > 4 languages demo).
> >
> > Chineese has few different writing systems.
> >
> > Buginese can use Lontara or Latin script.
> >
> >
> > There might be more of such cases.
> >
> > --
> > "My method is uncertain/ It's a mess but it's working" [F. Apple]
> >
> >
> > ___
> > talk mailing list
> > talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk
> >
> >  


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

Btw, this is already possible - Wikipedia servers let you access both
images and raw vector tiles, so if someone wants to do the client part, it
shouldn't be too hard.

On Fri, May 4, 2018, 11:11 Yuri Astrakhan  wrote:

> In reality, it is not impossible, or even that hard. If the vector tile is
> sent to the client, than the client can decide which language to render
> based on user preference.  The exact same code (it's all in JavaScript) can
> be used to decide the labeling. Caching would only improve, because instead
> of caching multiple raster tiles, it would only cache a single vector tile.
>
>  Mapbox.gl or open layers can both do this fairly efficiently.
>
>
>
> On Fri, May 4, 2018, 10:54 Mateusz Konieczny 
> wrote:
>
>>
>>
>>
>> 4. May 2018 07:36 by md...@xs4all.nl:
>>
>> On 2018-05-04 00:33, Joe Matazzoni wrote:
>>
>> No fallback is currently defined for Polish. We’ll be happy to
>> change that if you can show community consensus.
>>
>>
>> Community consensus? You mean a bunch of people who decide for the whole
>> country (many of them have no idea of the mechanisms behind it) what the
>> strategy is?
>> I'm sorry to say, but that can not be consensus. This needs to be able to
>> be configured at user level.
>>
>>
>> Server resources are not infinite, separate map cache for every user
>> would be probably unfeasible.
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

In reality, it is not impossible, or even that hard. If the vector tile is
sent to the client, than the client can decide which language to render
based on user preference.  The exact same code (it's all in JavaScript) can
be used to decide the labeling. Caching would only improve, because instead
of caching multiple raster tiles, it would only cache a single vector tile.

 Mapbox.gl or open layers can both do this fairly efficiently.



On Fri, May 4, 2018, 10:54 Mateusz Konieczny 
wrote:

>
>
>
> 4. May 2018 07:36 by md...@xs4all.nl:
>
> On 2018-05-04 00:33, Joe Matazzoni wrote:
>
> No fallback is currently defined for Polish. We’ll be happy to
> change that if you can show community consensus.
>
>
> Community consensus? You mean a bunch of people who decide for the whole
> country (many of them have no idea of the mechanisms behind it) what the
> strategy is?
> I'm sorry to say, but that can not be consensus. This needs to be able to
> be configured at user level.
>
>
> Server resources are not infinite, separate map cache for every user would
> be probably unfeasible.
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Mateusz Konieczny]

3. May 2018 22:49 by jmatazz...@wikimedia.org :


> I’d very much like to get your suggestions for a short list of Help links on 
> OSM—pages you think a user coming in to add multilingual names would find 
> useful. Also please send your thoughts about any information you think I 
> particularly should impart. 
>




1) Don't be afraid to ask (though, as usually, checking 


standards tutorials before doing that is a good idea) - 


see https://wiki.openstreetmap.org/wiki/Get_help 


for example of list of help pages/forums.




2) Note that OSM is stricter on copyright that Wikipedia or Wikidata.




It is OK to copy from any printed map or Google maps - it is even 


encouraged in Wikipedia[1] and it is utterly unacceptable in OSM[2].




It applies also to data that were copied from copyrighted databases into

Wikidata (some went through Wikipedias).





[1] https://en.wikipedia.org/wiki/Wikipedia:Obtaining_geographic_coordinates 



[2] https://wiki.openstreetmap.org/wiki/Copyright#Proprietary_data 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Mateusz Konieczny]

4. May 2018 07:36 by md...@xs4all.nl :


> On 2018-05-04 00:33, Joe Matazzoni wrote:
>> No fallback is currently defined for Polish. We’ll be happy to
>> change that if you can show community consensus.
>
> Community consensus? You mean a bunch of people who decide for the whole 
> country (many of them have no idea of the mechanisms behind it) what the 
> strategy is?
> I'm sorry to say, but that can not be consensus. This needs to be able to be 
> configured at user level.
>




Server resources are not infinite, separate map cache for every user would be 
probably unfeasible.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

Daniel, sure, I can give you the full scoop. Considering that I designed
and developed most of it, I do have some knowledge here. Big kudos to Max
Semenik for developing all the initial data queries and php work, all the
amazing UI done by Julien Girault, and the current maps engineering team
for a lot of good recent work, especially the multilingual support.

Summary:
* OSM data is imported a postgres DB with osm2pgsql (initially used default
schema, now migrating to ClearTables)
* Tilerator (Kartohterian's tile generation manager) is used to generate
vector tiles from SQL queries. One of the SQL output fields is a
JSON-formatted value with all the available languages  --  {"en":'"...,"
"fr":"...", ...}.  Mapnik creates vector tiles from the SQL output, and
babel decodes them, expands languages into multiple fields, and
recompresses them. Tilerator then stores them into Cassandra database
(WMF), but you could use any other storage, e.g. postgress or mbtiles.  So
as the result, tilerator's pipeline produces vector tiles with all
available languages.  No rendering is done yet.   Tilerator generates just
the  z0..z14.  My guesstimate is ~300GB, but these numbers could have
changed. Sadly, tiles tiles are not as optimized as what Open Map Tiles
produce, hence slightly oversized.
* When user requests a tile in a given language, vector tile is loaded from
storage, modified on the fly by using babel - unpacks the tile and this
time choose just one language. The tile is repacked and passed to mapnik
for rendering. The resulting tile is not stored on disk, but it does get
cached by Varnish.

So as you can see, there is nearly no difference between one language and
unlimited number of languages with this approach - except for the slightly
slower generation and rendering, and bigger cache fragmentation.  See
various Kartotherian repos in github for more information.


-- Yuri  / @nyuriks

On Fri, May 4, 2018 at 4:14 AM Daniel Koć  wrote:

> W dniu 04.05.2018 o 02:30, Yuri Astrakhan pisze:
> > Daniel, the only real difference between serving every available
> > language and serving just one is cache fragmentation, and that's may
> > be different in your case.
>
> Sure, but you're talking about just one link of the chain. The WMF
> vector tiles are produced from a database, rasterized somewhere down the
> line, then probably written on the disk, served, cached etc, so it's not
> that simple.
>
> That's why I ask about the bottom line in this specific case. There are
> many factors which will be different (like how complex the style is, for
> example), but it's good to estimate the order of magnitude at least. But
> all the details are also interesting to me.
>
> --
> "My method is uncertain/ It's a mess but it's working" [F. Apple]
>
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Jo]

Japanese has 3 writing systems.

Jo

2018-05-04 1:10 GMT+02:00 Daniel Koć :

> W dniu 04.05.2018 o 00:33, Joe Matazzoni pisze:
>
> Here is the list of languages Wikimedia supports. https://en.
> wikipedia.org/wiki/Special:SiteMatrix
> 
>  .
> As to the toolchain, if you ask that question on the project page, some of
> our engineers will be able to respond.  Be sure to explain why you’re
> asking, so we can answer you fully.
>
>
> It's not exactly what interests me:
>
> a) do you want to support all these languages (not how many of them are
> there)?
>
> b) how many server resources do you need for rendering  languages that
> you want to support (not the software stack used)?
>
> Do you know it or should I ask the engineers anyway?
>
> That is a very interesting edge case we hadn’t thought of.  We’ll have to
> look into that one; our lead engineer just wrote a ticket to investigate:
> https://phabricator.wikimedia.org/T193815
> .
>  I know about Serbian. What other languages do this?
>
>
> Belorusian has two writing systems (see http://openstreetmap.by for 4
> languages demo).
>
> Chineese has few different writing systems.
>
> Buginese can use Lontara or Latin script.
>
>
> There might be more of such cases.
>
> --
> "My method is uncertain/ It's a mess but it's working" [F. Apple]
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Maarten Deen]

On 2018-05-04 00:33, Joe Matazzoni wrote:

Hi Daniel,
I’ll look for your contributions on the project talk page. Thanks.
Meanwhile, here are answers to your questions:


1. The localized maps lack fallback rules (I'm speaking of Polish



language at least). I would ask for English as a fallback for maps
in



Polish, but I don't know where should it be requested or
configured? Is



this list the right place?:


No fallback is currently defined for Polish. We’ll be happy to
change that if you can show community consensus.


Community consensus? You mean a bunch of people who decide for the whole 
country (many of them have no idea of the mechanisms behind it) what the 
strategy is?
I'm sorry to say, but that can not be consensus. This needs to be able 
to be configured at user level.


Maarten

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Daniel Koć]

W dniu 04.05.2018 o 02:30, Yuri Astrakhan pisze:
> Daniel, the only real difference between serving every available
> language and serving just one is cache fragmentation, and that's may
> be different in your case.

Sure, but you're talking about just one link of the chain. The WMF
vector tiles are produced from a database, rasterized somewhere down the
line, then probably written on the disk, served, cached etc, so it's not
that simple.

That's why I ask about the bottom line in this specific case. There are
many factors which will be different (like how complex the style is, for
example), but it's good to estimate the order of magnitude at least. But
all the details are also interesting to me.

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

Daniel, the only real difference between serving every available language
and serving just one is cache fragmentation, and that's may be different in
your case.

The vector tiles get pre-generated the same way, and they simply contain
all languages instead of one. Disk space wise, the difference is
inconsequential.

Kartotherian dynamically picks the language during the rendering. There is
some (not yet measured, but hopefully very small) performance penalty doing
dynamic language picking, but that should not be a big impact in case of a
good caching in front of the rendering.  I would recommend simply putting a
few Varnish boxes in front of your rendering servers if you are ok with the
maps being a bit stale.

On Fri, May 4, 2018 at 2:32 AM Daniel Koć  wrote:

> W dniu 04.05.2018 o 01:17, Joe Matazzoni pisze:
>
> > b) how many server resources do you need for rendering  languages
> > that you want to support (not the software stack used)?
> >
> > ?? Are you asking about OSM resources or WMF resources?
>
> I mean WMF resources.
>
> I'm asking because we want to have localized maps in OSM too and
> rendering just one (default) raster style eats most of the resources of
> 4 servers. This would not work for (let's say) 300 language versions, of
> course.
>
> We refresh default rendering in minutes usually, but even if we use more
> relaxed times for localized maps, it's still too much - with 1 language
> per day we would refresh them all once a year...
>
> We think about vector style migration lately, so it might help here, but
> I don't think this will work like 100x times faster.
>
> --
> "My method is uncertain/ It's a mess but it's working" [F. Apple]
>
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Pierre Béland]

I have seen a lot of these images and I also believe that we wont need these 
static images if we can interface properly between OSM and Wikipedia. We just 
need the recipes for dynamic connections between the two systems :)

 
Pierre 
 

Le jeudi 3 mai 2018 19 h 43 min 38 s HAE, Joe Matazzoni 
 a écrit :  
 
 I found another OSM page directed at OSM-Wikimedia collaboration [1]. This one 
encourages OSM users to add images of OSM maps to Wikimedia wikis as static 
graphics. As such, I wonder if advances in placing dynamic dynamic maps in 
wikis (though mapframe and maplink has made this page somewhat out of date? I 
don’t feel that I have the standing in the OSM community to update such a page…
And please remember to suggest a short list of OSM Help pages that will be 
useful for Wikimedians coming to OSM for the first time to add multilingual 
names. Thanks!  
Joe


[1] https://wiki.openstreetmap.org/wiki/Collaboration_with_Wikipedia

_


Joe Matazzoni 
 Product Manager, Collaboration
Wikimedia Foundation, San Francisco

"Imagine a world in which every single human being can freely share in the sum 
of all knowledge." 

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk
  ___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Pierre Béland]

Hi Joe

These features are promizing. My observations will be focused on the 
interrelations between OSM and Wikipeda and my perception that OSM contributors 
need more documentation to support Wikipedians.

I was contacted recently by a  Wikipedian who talked to me about the maplink 
feature and was looking how to represent a river (watershed or other 
possibilities). From our exchanges, we saw different possibilities but I could 
not find what to do on the OSM side to use these features. It says that we can 
both represent polygons or lines. Adding wikidata and waiting up to four days, 
I had no success.  Good examples that are working would help.  I have a 
Wikipeda account and also tried to use these features in a wikipedia page using 
mapframe but with no success.

Should we simply add a wikidata and wikipedia will take care. Should we define 
a mapframe in a wikipedia page? Or the two options are available?

The example below shows an OSM relation for a polygon that is well represented 
in Wikipedia.https://fr.wikipedia.org/wiki/R%C3%A9servoir_La_Grande_4#/maplink/0

But adding a wikidata tag to a OSM waterway riverbank about four days ago, I 
only see a  node represented on the wikipedia maplink map. 
https://fr.wikipedia.org/wiki/Rivi%C3%A8re_L'Assomption#/maplink/0
We need more infos on what type of OSM features can be represented in 
Wikipedia. Also, should the wikidata be unique or not in OSM. For example, if I 
represent a municipality with both a relation for the polygon borders and a 
node for the OSM place, where should I place the wikidata? On the relation? on  
the node? On both? Or two different wikidatga iD's? It depends if I want to 
represent the polygon or the node or both?

Territories are also interesting to represent. We often see images in Wikipedia 
but how to make a link to an OSM object in Wikipedia? See for example 
Longueuil, Québec who has two different wikidata for polygon and node. 
Wikipedia maplink shows a node, but not sure this is related to the node 
wikidata.
https://fr.wikipedia.org/wiki/Longueuil#/maplink/0https://www.openstreetmap.org/relation/6948543
https://www.openstreetmap.org/node/252025539 
Pierre 
 

Le jeudi 3 mai 2018 16 h 53 min 21 s HAE, Joe Matazzoni 
 a écrit :  
 
 Hello OpenStreetMap community,I’m the product manager of the Wikimedia 
Foundation (WMF) Collaboration Team. We’ve been working on project recently 
called Map Improvements 2018 [1] that some of you will find interesting. As 
most of you probably know, WMF maps are powered by OSM data. The most 
significant new feature that we’ll be releasing very soon as part of this 
project is map “internationalization”—which means that’s WMF maps will display 
in the language of the user, rather than of the territory mapped. I wrote a 
recent post describing this feature and how it works [2]. 
We’re also about to spread our embedded maps capability (“mapframe”) to 
hundreds of Wikipedias that don’t have the feature now. The 
internationalization release will follow soon after. These should be welcome 
developments for the OSM community, I hope, since they will put OSM-powered 
maps in front of many millions of new users. 
We don’t anticipate that these new maps will put any strain on OSM performance. 
The impact I do foresee—and hope for—is that the new exposure of multilingual 
map data will inspire many more Wikimedians to contribute to OSM. This is 
likely to happen when users start to see, as they will for the first time, that 
names in their language for some features and places are not available.  
I’m writing today to let you know that these changes—and possibly these new 
contributors—are coming, and to ask for any guidance you think I should pass on 
to Wikimedians who might like to contribute to OSM. We plan to write a Help 
page on our end that will pass on some basic advice. And I will certainly link 
to relevant OSM Help pages, including this"Welcome to Wikipedia users” [3] 
page. I’d very much like to get your suggestions for a short list of Help links 
on OSM—pages you think a user coming in to add multilingual names would find 
useful. Also please send your thoughts about any information you think I 
particularly should impart. 
Please post your thoughts and ideas to the project talk page [4]. Thanks for 
your help, and for providing the valuable service you do to Wikimedia 
contributors and readers around the world.  

[1]https://www.mediawiki.org/wiki/Map_improvements_2018[2] 
https://www.mediawiki.org/wiki/Map_improvements_2018#April_18,_2018,_Special_Update_on_Map_Internationalization[3]
 https://wiki.openstreetmap.org/wiki/Welcome_to_Wikipedia_users[4] 
https://www.mediawiki.org/wiki/Talk:Map_improvements_2018
_

Joe Matazzoni 
 Product Manager, Collaboration
Wikimedia Foundation, San Francisco

"Imagine a world in which every single human being can freely share in the sum 
of all knowledge." 



___
talk mailing l

Re: [OSM-talk] OSM and new Wikipedia map features

[Joe Matazzoni]

I found another OSM page directed at OSM-Wikimedia collaboration [1]. This one 
encourages OSM users to add images of OSM maps to Wikimedia wikis as static 
graphics. As such, I wonder if advances in placing dynamic dynamic maps in 
wikis (though mapframe and maplink has made this page somewhat out of date? I 
don’t feel that I have the standing in the OSM community to update such a page…

And please remember to suggest a short list of OSM Help pages that will be 
useful for Wikimedians coming to OSM for the first time to add multilingual 
names. Thanks!  

Joe


[1] https://wiki.openstreetmap.org/wiki/Collaboration_with_Wikipedia 

_

Joe Matazzoni 
Product Manager, Collaboration
Wikimedia Foundation, San Francisco


"Imagine a world in which every single human being can freely share in the sum 
of all knowledge." 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Daniel Koć]

W dniu 04.05.2018 o 01:17, Joe Matazzoni pisze:

> b) how many server resources do you need for rendering  languages
> that you want to support (not the software stack used)?
>
> ?? Are you asking about OSM resources or WMF resources?

I mean WMF resources.

I'm asking because we want to have localized maps in OSM too and
rendering just one (default) raster style eats most of the resources of
4 servers. This would not work for (let's say) 300 language versions, of
course.

We refresh default rendering in minutes usually, but even if we use more
relaxed times for localized maps, it's still too much - with 1 language
per day we would refresh them all once a year...

We think about vector style migration lately, so it might help here, but
I don't think this will work like 100x times faster.

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Daniel Koć]

W dniu 04.05.2018 o 00:33, Joe Matazzoni pisze:

> Here is the list of languages Wikimedia
> supports. https://en.wikipedia.org/wiki/Special:SiteMatrix
> 
>  .
> As to the toolchain, if you ask that question on the project page,
> some of our engineers will be able to respond.  Be sure to explain why
> you’re asking, so we can answer you fully.

It's not exactly what interests me:

a) do you want to support all these languages (not how many of them are
there)?

b) how many server resources do you need for rendering  languages
that you want to support (not the software stack used)?

Do you know it or should I ask the engineers anyway?

> That is a very interesting edge case we hadn’t thought of.  We’ll have
> to look into that one; our lead engineer just wrote a ticket to
> investigate: https://phabricator.wikimedia.org/T193815
> .
>  I know about Serbian. What other languages do this?

Belorusian has two writing systems (see http://openstreetmap.by for 4
languages demo).

Chineese has few different writing systems.

Buginese can use Lontara or Latin script.


There might be more of such cases.

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Daniel Koć]

W dniu 03.05.2018 o 23:20, Yuri Astrakhan pisze:

> I have posted in another threads about this, proposing
> "default_language" tag to be added to the admin (or smaller) regions
> to solve this. 

I like this proposition. I was talking about "official_language", but
they might be added where they are known and defined, yet most of the
time "default_language" would be the best solution.

But we need to distinguish two different cases: hint for debugging
"name" tag (or any other, like "alt_name", for that matter) and real set
of default languages, for example:

name=België / Belgique / Belgien
name:default_language=nl / fr / de
default_language=de;fr;nl

You can analyze "name" (which might be specially crafted, for example to
be shorter) or just glue proper values like name:de, name:fr and name:nl
somehow, whatever suits your needs better.

We should also think of different scripts or conventions used with the
same language. Example:

name=Norge
name:default_language=nb

name:nb=Norge
name:nn=Noreg
default_language=nb;nn

Another example (I have to use some guess work):

name=中国
name:default_language=zh

name:zh=中国
name:zh-classical=中國
name:zh-hans=中国
name:zh-hant=中國
name:zh-min-nan=Tiong-hôa
name:zh-simplified=中国
name:zh-traditional=中國
name:zh-yue=中國
name:zh_pinyin=Zhōngguó
default_language=zh

(well, probably...)

> Copying the rules:
>
> * Use the largest possible admin region to set the "default_language"
> tag to a single language code.  "default_language" does not mean the
> official language of the region. It only specifies the language of the
> "name" tag.
> * A region may contain a sub-region with a different default_language.
> * If a region uses mixed languages in all of its name tags, eg. "[name
> in en] - [name in zh]", set default_language="en - zh".  Try to keep
> it to a somewhat parsable value to help data consumers.
> * In some rare cases, additional non-admin regions might be required
> for the default_language.  Try to avoid it if possible.

I would also add "default_language=none". It's not needed until you make
the cascading definitions (inheritance). When you don't know or it's
just hard to get right, it's good to "switch off" and fallback to what
we have now.

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Joe Matazzoni]

Hi Daniel,
I’ll look for your contributions on the project talk page. Thanks. Meanwhile, 
here are answers to your questions:

> 1. The localized maps lack fallback rules (I'm speaking of Polish

> language at least). I would ask for English as a fallback for maps in

> Polish, but I don't know where should it be requested or configured? Is

> this list the right place?:


No fallback is currently defined for Polish. We’ll be happy to change that if 
you can show community consensus. Make sure your community understands that 
maps have a separate fallback list from the general language fallback list for 
the wiki, where the fallback controls things like what language UI messages are 
shown in. The two lists are identical at the moment, but having two separate 
lists means that you can change the way maps behave without having to change 
behavior for the wiki in general. So that should make getting consensus easier. 
When you have consensus, leave a request with a link to the discussion on the 
maps project page, or fill in a phabricator ticket and tag it to both “maps” 
and "Collaboration-Feature-Rollouts (Collaboration-Maps)”.


> 2. How many languages do you want to support in total and what hardware

> resources are needed for that using your toolchain?

Here is the list of languages Wikimedia supports. 
https://en.wikipedia.org/wiki/Special:SiteMatrix 

 . As to the toolchain, if you ask that question on the project page, some of 
our engineers will be able to respond.  Be sure to explain why you’re asking, 
so we can answer you fully. 

> 3. What about the same language using different scripts, do you plan to

> support them all?


That is a very interesting edge case we hadn’t thought of.  We’ll have to look 
into that one; our lead engineer just wrote a ticket to investigate: 
https://phabricator.wikimedia.org/T193815 
.
  I know about Serbian. What other languages do this? 

Best,
Joe
_

Joe Matazzoni 
Product Manager, Collaboration
Wikimedia Foundation, San Francisco


"Imagine a world in which every single human being can freely share in the sum 
of all knowledge." 




> On May 3, 2018, at 2:50 PM, Daniel Koć  wrote:
> 
> W dniu 03.05.2018 o 22:49, Joe Matazzoni pisze:
> 
>> Please post your thoughts and ideas to the project talk page [4].
>> Thanks for your help, and for providing the valuable service you do to
>> Wikimedia contributors and readers around the world. 
> 
> I will try to put my thoughts there.
> 
> However I also have some questions for you:
> 
> 1. The localized maps lack fallback rules (I'm speaking of Polish
> language at least). I would ask for English as a fallback for maps in
> Polish, but I don't know where should it be requested or configured? Is
> this list the right place?:
> 
> https://github.com/kartotherian/babel/blob/master/lib/fallbacks.json
> 
> 2. How many languages do you want to support in total and what hardware
> resources are needed for that using your toolchain?
> 
> 3. What about the same language using different scripts, do you plan to
> support them all?
> 
> -- 
> "My method is uncertain/ It's a mess but it's working" [F. Apple]
> 
> 
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

I cannot speak for WMF, only about the actual Kartotherian stack behind it,
and the way they are currently using it:

On Fri, May 4, 2018 at 12:52 AM Daniel Koć  wrote:

>
> 1. The localized maps lack fallback rules (I'm speaking of Polish
> language at least). I would ask for English as a fallback for maps in
> Polish, but I don't know where should it be requested or configured? Is
> this list the right place?:
>
> https://github.com/kartotherian/babel/blob/master/lib/fallbacks.json
>
> See also how the name tag is picked here:
https://github.com/kartotherian/babel/blob/master/lib/LanguagePicker.js#L106
It has went through several revisions recently, so this is somewhat in a
flux.

2. How many languages do you want to support in total and what hardware
> resources are needed for that using your toolchain?
>

Kartotherian itself can handle unlimited number of languages.  Vector tiles
can simply hold every possible language, and babel picks the one you want
based on the above heuristic.   My understanding is that currently WMF does
not filter out any languages, nor does it plan to, so any lang=xx would
work, where xx comes directly from the name:xx tags.


> 3. What about the same language using different scripts, do you plan to
> support them all?
>
> Babel does not actually know much about the "language". It looks at the
language codes, trying to match it to the fallbacks, and uses heuristic
when fallback fails.   E.g. it should be possible to simply have two
language codes for Serbian in Latin & Cyrillic, and say that if one is
available when the other is not, to fallback.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Daniel Koć]

W dniu 03.05.2018 o 22:49, Joe Matazzoni pisze:

> Please post your thoughts and ideas to the project talk page [4].
> Thanks for your help, and for providing the valuable service you do to
> Wikimedia contributors and readers around the world. 

I will try to put my thoughts there.

However I also have some questions for you:

1. The localized maps lack fallback rules (I'm speaking of Polish
language at least). I would ask for English as a fallback for maps in
Polish, but I don't know where should it be requested or configured? Is
this list the right place?:

https://github.com/kartotherian/babel/blob/master/lib/fallbacks.json

2. How many languages do you want to support in total and what hardware
resources are needed for that using your toolchain?

3. What about the same language using different scripts, do you plan to
support them all?

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

P.S. example - this library [1] might be a good fit (any other
suggestions?). It does universal transliteration, yet even the author
writes this:

"transliteration supports almost all common languages whereas there might
be quirks in some specific languages. For example, Kanji characters in
Japanese will be transliterated as Chinese Pinyin. I couldn't find a better
way to distinguish Chinese Hanzi and Japanese Kanji. So if you would like
to romanize Japanese Kanji, please consider kuroshiro."

[1]: https://www.npmjs.com/package/transliteration#caveats

On Fri, May 4, 2018 at 12:20 AM Yuri Astrakhan 
wrote:

> Christoph, I agree that this would be an awesome improvement, yet I think
> there is a problem to implement it. Most languages have their own
> transliteration rules, so transliterating "name" tag without the knowledge
> of its language will produce a lot of incorrect names.
>
> I have posted in another threads about this, proposing "default_language"
> tag to be added to the admin (or smaller) regions to solve this.  Copying
> the rules:
>
> * Use the largest possible admin region to set the "default_language" tag
> to a single language code.  "default_language" does not mean the official
> language of the region. It only specifies the language of the "name" tag.
> * A region may contain a sub-region with a different default_language.
> * If a region uses mixed languages in all of its name tags, eg. "[name in
> en] - [name in zh]", set default_language="en - zh".  Try to keep it to a
> somewhat parsable value to help data consumers.
> * In some rare cases, additional non-admin regions might be required for
> the default_language.  Try to avoid it if possible.
>
>
> On Fri, May 4, 2018 at 12:09 AM Christoph Hormann  wrote:
>
>> On Thursday 03 May 2018, Joe Matazzoni wrote:
>> > [...]
>> >
>> > We don’t anticipate that these new maps will put any strain on OSM
>> > performance. The impact I do foresee—and hope for—is that the new
>> > exposure of multilingual map data will inspire many more Wikimedians
>> > to contribute to OSM. This is likely to happen when users start to
>> > see, as they will for the first time, that names in their language
>> > for some features and places are not available.
>>
>> The first and most fundamental thing you should do is add
>> automatic transliteration as a fallback for multilingual names.
>> Otherwise people will inevitably add tons of non-verifiable
>> transliterated names in a misguided attempt improve the map.
>>
>> --
>> Christoph Hormann
>> http://www.imagico.de/
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Yuri Astrakhan]

Christoph, I agree that this would be an awesome improvement, yet I think
there is a problem to implement it. Most languages have their own
transliteration rules, so transliterating "name" tag without the knowledge
of its language will produce a lot of incorrect names.

I have posted in another threads about this, proposing "default_language"
tag to be added to the admin (or smaller) regions to solve this.  Copying
the rules:

* Use the largest possible admin region to set the "default_language" tag
to a single language code.  "default_language" does not mean the official
language of the region. It only specifies the language of the "name" tag.
* A region may contain a sub-region with a different default_language.
* If a region uses mixed languages in all of its name tags, eg. "[name in
en] - [name in zh]", set default_language="en - zh".  Try to keep it to a
somewhat parsable value to help data consumers.
* In some rare cases, additional non-admin regions might be required for
the default_language.  Try to avoid it if possible.


On Fri, May 4, 2018 at 12:09 AM Christoph Hormann  wrote:

> On Thursday 03 May 2018, Joe Matazzoni wrote:
> > [...]
> >
> > We don’t anticipate that these new maps will put any strain on OSM
> > performance. The impact I do foresee—and hope for—is that the new
> > exposure of multilingual map data will inspire many more Wikimedians
> > to contribute to OSM. This is likely to happen when users start to
> > see, as they will for the first time, that names in their language
> > for some features and places are not available.
>
> The first and most fundamental thing you should do is add
> automatic transliteration as a fallback for multilingual names.
> Otherwise people will inevitably add tons of non-verifiable
> transliterated names in a misguided attempt improve the map.
>
> --
> Christoph Hormann
> http://www.imagico.de/
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM and new Wikipedia map features

[Christoph Hormann]

On Thursday 03 May 2018, Joe Matazzoni wrote:
> [...]
>
> We don’t anticipate that these new maps will put any strain on OSM
> performance. The impact I do foresee—and hope for—is that the new
> exposure of multilingual map data will inspire many more Wikimedians
> to contribute to OSM. This is likely to happen when users start to
> see, as they will for the first time, that names in their language
> for some features and places are not available.

The first and most fundamental thing you should do is add 
automatic transliteration as a fallback for multilingual names.  
Otherwise people will inevitably add tons of non-verifiable 
transliterated names in a misguided attempt improve the map.

-- 
Christoph Hormann
http://www.imagico.de/

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk