Re: [Server-devel] [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

[Jérôme Gagnon-Voyer]

Hi all

I wanted to do some testing of the search solutions on some kind of
hardware/software that would look similar to what will be deployed in the
field (to measure the performance impact).

Got this idea from this post where :
http://www.openstreetmap.org/user/SimonPoole/diary/34857

Should I be testing with XSCE 5.1? XSCE 5.5 beta? Should I get some
hardware like a Pi2, or just get some VM? Any VM recommendation?

Thanks

On Wed, Jun 17, 2015 at 3:15 PM, Tim Moody  wrote:

> Thanks, this is very useful information.  I didn't realize that slippy
> maps is a strategy and leaflet is a library for it that OSM itself uses.
>
>
>
> A few questions.
>
>
>
> Which of the filtered datasets did you use for populated places?
>
>
>
> Are they from
> http://wiki.openstreetmap.org/wiki/Planet.osm#Country_and_area_extracts ?
>
>
>
> Is the directory structure such that separately rendered tiles for
> different regions can be merged by over-copying?
>
>
>
> I did some sampling.
>
>
>
> In Manhattan the maximum zoom is
> http://xsce/iiab/maps/tile/15/9649/12318.png so level 15 (16th)
>
>
>
> In Northern Ontario the maximum zoom is
> http://xsce/iiab/maps/tile/13/2048/2704.png level 13 (14th)
>
>
>
> In the middle of the North Pacific
> http://xsce/iiab/maps/tile/14/1088/6190.png a surprising level 14
>
>
>
> Moving slightly left I get http://xsce/iiab/maps/tile/14/1080/6191.png so
> I'm not quite clear on the generic blue tile point.
>
>
>
> I'm also not clear on the point about the meta tiles.  From what I read in
> http://wiki.openstreetmap.org/wiki/Meta_tiles , meta files just
> concatenate the smaller tiles, so I'm not clear on how that saves space.  I
> do understand that it greatly reduces the number of nodes in the file
> system making retrieving them viable.
>
>
>
> I looked at space usage by zoom level on the current IIAB OSM:
>
>
>
> Level
>
> Size (in K)
>
> Cumulative (in K)
>
> Cumulative (in G)
>
> 0
>
> 60
>
> 60
>
> 0
>
> 1
>
> 84
>
> 144
>
> 0
>
> 2
>
> 152
>
> 296
>
> 0
>
> 3
>
> 300
>
> 596
>
> 0
>
> 4
>
> 1,004
>
> 1,600
>
> 0
>
> 5
>
> 2,800
>
> 4,400
>
> 0
>
> 6
>
> 9,064
>
> 13,464
>
> 0
>
> 7
>
> 26,328
>
> 39,792
>
> 0
>
> 8
>
> 86,280
>
> 126,072
>
> 0
>
> 9
>
> 255,576
>
> 381,648
>
> 0
>
> 10
>
> 859,540
>
> 1,241,188
>
> 1
>
> 11
>
> 2,233,136
>
> 3,474,324
>
> 3
>
> 12
>
> 5,905,172
>
> 9,379,496
>
> 9
>
> 13
>
> 17,480,260
>
> 26,859,756
>
> 27
>
> 14
>
> 43,947,380
>
> 70,807,136
>
> 71
>
> 15
>
> 35,454,488
>
> 106,261,624
>
> 106
>
>
>
> So 10G gets 13 levels.
>
>
>
>
>
> *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com]
> *On Behalf Of *Braddock Gaskill
> *Sent:* Monday, June 15, 2015 8:39 PM
> *To:* Internet In a Box Working Group
> *Cc:* xsce-devel; Unleash Kids!; Jaakko Helleranta; server-devel; iaep;
> Community Support Volunteers -- who help respond to help AT laptop.org
> *Subject:* Re: [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking
> OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>
>
>
> In IIAB we use a dataset of populated places to render at the highest
> resolution level, and render the rest of the world at a slightly lower
> resolution.  Our highest resolution is high enough to see individual
> buildings.  Our total dataset of rendered tiles for the world with detailed
> populated places coverage is 100GB.
>
>
>
> Another optimization we made for IIAB that contributed considerable
> dataset size reduction and helped with filesystem efficiency - we use the
> OSM "meta" tiles, which normally store 64 tile png images, but we modify
> them to store 4096 png images per file, and we made an optimization so that
> identical tiles are removed (think of all the blue of the ocean cover 75%
> of the earth).
>
>
>
> We use Leaflet as our javascript web front end, and it works quite well.
>
>
>
> -braddock
>
>
>
>
>
> On Sun, Jun 14, 2015 at 11:24 PM, Nick Doiron  wrote:
>
> I don't think there's any technical issues with rendering the world at 10
> and specific countries at 16, other than the human knowing where they can
> and cannot zoom
>
>
>
> -- Nick
>
>
>
> On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody  wrote:
>
> couple of observations:
>
>
>
> As expected, the new tiles have a lot more detail.
>
>
>
> There are more levels of zoom in the new ones.
>
>
>
> Some names have changed - the old map had Lalitpur and the new one has
> Patan (both are used)
>
>
>
> I don't see any boxes for unprintable characters, but there is a lot less
> Devanagari. (Google maps has more)
>
>
>
> Is it possible to merge individually generated regional tiles?  for
> example if you rendered India and Nepal separately would you get both?
>
>
>
> What happens if you render the world at level 10 and then specific
> countries at 16?
>
>
>
> *From:* xsce-de...@googlegroups.com [mailto:xsce-de...@googlegroups.com] *On
> Behalf Of *Anish Mangal
> *Sent:* Saturday, June 13, 2015 10:06 PM
> *To:* Jérôme Gagnon-Voyer
> *Cc:* xsce-devel; Community Support Volunt

Re: [Server-devel] [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

[Tim Moody]

Thanks, this is very useful information.  I didn't realize that slippy maps is 
a strategy and leaflet is a library for it that OSM itself uses.

 

A few questions.

 

Which of the filtered datasets did you use for populated places? 

 

Are they from 
http://wiki.openstreetmap.org/wiki/Planet.osm#Country_and_area_extracts ?

 

Is the directory structure such that separately rendered tiles for different 
regions can be merged by over-copying?

 

I did some sampling.

 

In Manhattan the maximum zoom is  http://xsce/iiab/maps/tile/15/9649/12318.png 
so level 15 (16th)

 

In Northern Ontario the maximum zoom is 
http://xsce/iiab/maps/tile/13/2048/2704.png level 13 (14th)

 

In the middle of the North Pacific http://xsce/iiab/maps/tile/14/1088/6190.png 
a surprising level 14

 

Moving slightly left I get http://xsce/iiab/maps/tile/14/1080/6191.png so I'm 
not quite clear on the generic blue tile point.

 

I'm also not clear on the point about the meta tiles.  From what I read in 
http://wiki.openstreetmap.org/wiki/Meta_tiles , meta files just concatenate the 
smaller tiles, so I'm not clear on how that saves space.  I do understand that 
it greatly reduces the number of nodes in the file system making retrieving 
them viable.

 

I looked at space usage by zoom level on the current IIAB OSM:

 


Level

Size (in K)

Cumulative (in K)

Cumulative (in G)



0

60

60

0


1

84

144

0


2

152

296

0


3

300

596

0


4

1,004

1,600

0


5

2,800

4,400

0


6

9,064

13,464

0


7

26,328

39,792

0


8

86,280

126,072

0


9

255,576

381,648

0


10

859,540

1,241,188

1


11

2,233,136

3,474,324

3


12

5,905,172

9,379,496

9


13

17,480,260

26,859,756

27


14

43,947,380

70,807,136

71


15

35,454,488

106,261,624

106

 

So 10G gets 13 levels.

 

 

From: unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com] On 
Behalf Of Braddock Gaskill
Sent: Monday, June 15, 2015 8:39 PM
To: Internet In a Box Working Group
Cc: xsce-devel; Unleash Kids!; Jaakko Helleranta; server-devel; iaep; Community 
Support Volunteers -- who help respond to help AT laptop.org
Subject: Re: [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap 
Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

 

In IIAB we use a dataset of populated places to render at the highest 
resolution level, and render the rest of the world at a slightly lower 
resolution.  Our highest resolution is high enough to see individual buildings. 
 Our total dataset of rendered tiles for the world with detailed populated 
places coverage is 100GB.

 

Another optimization we made for IIAB that contributed considerable dataset 
size reduction and helped with filesystem efficiency - we use the OSM "meta" 
tiles, which normally store 64 tile png images, but we modify them to store 
4096 png images per file, and we made an optimization so that identical tiles 
are removed (think of all the blue of the ocean cover 75% of the earth).

 

We use Leaflet as our javascript web front end, and it works quite well.

 

-braddock

 

 

On Sun, Jun 14, 2015 at 11:24 PM, Nick Doiron mailto:ndoi...@mapmeld.com> > wrote:

I don't think there's any technical issues with rendering the world at 10 and 
specific countries at 16, other than the human knowing where they can and 
cannot zoom

 

-- Nick

 

On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody mailto:t...@timmoody.com> > wrote:

couple of observations:

 

As expected, the new tiles have a lot more detail.

 

There are more levels of zoom in the new ones.

 

Some names have changed - the old map had Lalitpur and the new one has Patan 
(both are used)

 

I don't see any boxes for unprintable characters, but there is a lot less 
Devanagari. (Google maps has more)

 

Is it possible to merge individually generated regional tiles?  for example if 
you rendered India and Nepal separately would you get both?

 

What happens if you render the world at level 10 and then specific countries at 
16?

 

From: xsce-de...@googlegroups.com   
[mailto:xsce-de...@googlegroups.com  ] On 
Behalf Of Anish Mangal
Sent: Saturday, June 13, 2015 10:06 PM
To: Jérôme Gagnon-Voyer
Cc: xsce-devel; Community Support Volunteers -- who help respond to help AT 
laptop.org  ; Unleash Kids!; server-devel; iaep; Internet In 
a Box Working Group; Jaakko Helleranta
Subject: Re: [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline 
- DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

 

Hi,

Following from the skype call this week, I uploaded Nepal's OSM data in 
postgres and render tiles to see performance, and as expected, everything is 
blazing fast. 

The pbf file went into the postgress in less than 5 minutes, and tiles are 
being rendered pretty fast as well.

You can check it yourself by going to 
http://home.braddock.com:28112/osm/slippymap.html

* Select Mapnik

* Zoo

Re: [Server-devel] [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

[Braddock Gaskill]

In IIAB we use a dataset of populated places to render at the highest
resolution level, and render the rest of the world at a slightly lower
resolution.  Our highest resolution is high enough to see individual
buildings.  Our total dataset of rendered tiles for the world with detailed
populated places coverage is 100GB.

Another optimization we made for IIAB that contributed considerable dataset
size reduction and helped with filesystem efficiency - we use the OSM
"meta" tiles, which normally store 64 tile png images, but we modify them
to store 4096 png images per file, and we made an optimization so that
identical tiles are removed (think of all the blue of the ocean cover 75%
of the earth).

We use Leaflet as our javascript web front end, and it works quite well.

-braddock


On Sun, Jun 14, 2015 at 11:24 PM, Nick Doiron  wrote:

> I don't think there's any technical issues with rendering the world at 10
> and specific countries at 16, other than the human knowing where they can
> and cannot zoom
>
> -- Nick
>
> On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody  wrote:
>
>> couple of observations:
>>
>>
>>
>> As expected, the new tiles have a lot more detail.
>>
>>
>>
>> There are more levels of zoom in the new ones.
>>
>>
>>
>> Some names have changed - the old map had Lalitpur and the new one has
>> Patan (both are used)
>>
>>
>>
>> I don't see any boxes for unprintable characters, but there is a lot less
>> Devanagari. (Google maps has more)
>>
>>
>>
>> Is it possible to merge individually generated regional tiles?  for
>> example if you rendered India and Nepal separately would you get both?
>>
>>
>>
>> What happens if you render the world at level 10 and then specific
>> countries at 16?
>>
>>
>>
>> *From:* xsce-de...@googlegroups.com [mailto:xsce-de...@googlegroups.com] *On
>> Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 10:06 PM
>> *To:* Jérôme Gagnon-Voyer
>> *Cc:* xsce-devel; Community Support Volunteers -- who help respond to
>> help AT laptop.org; Unleash Kids!; server-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* Re: [XSCE] RE: [UKids] Re: [support-gang] Taking
>> OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Hi,
>>
>> Following from the skype call this week, I uploaded Nepal's OSM data in
>> postgres and render tiles to see performance, and as expected, everything
>> is blazing fast.
>>
>> The pbf file went into the postgress in less than 5 minutes, and tiles
>> are being rendered pretty fast as well.
>>
>> You can check it yourself by going to
>> http://home.braddock.com:28112/osm/slippymap.html
>>
>> * Select Mapnik
>>
>> * Zoom out and center over Nepal
>>
>> * Select Local tiles
>>
>> * Zoom in
>>
>> I don't know what this "proves" as the bottleneck still is doing this for
>> the entire planet, which we need to figure out a way for anyway.
>>
>> Best,
>>
>> Anish
>>
>>
>>
>>
>>
>> On Sun, Jun 14, 2015 at 6:02 AM, Anish Mangal  wrote:
>>
>> Hi Jérôme,
>>
>> I incorporated one of your comments; as for the other I think Timm, Nick
>> would be better suited for the discussion (python backend v/s frontend js).
>> :-)
>>
>> Best,
>>
>> Anish
>>
>>
>>
>> On Sun, Jun 14, 2015 at 3:22 AM, Jérôme Gagnon-Voyer 
>> wrote:
>>
>> Hi Anish
>>
>>
>>
>> Great document. I've added 2 comments to the document, feel free to
>> incorporate into the document if that makes sense.
>>
>>
>>
>> Also added a TODO for myself to do more research about the various search
>> solutions.
>>
>> I've used Nominatim in the past (on the client side, not the server
>> infrastructure) and it was overall very good, but I'll want to know more
>> about the other ones.
>>
>>
>>
>> On Sat, Jun 13, 2015 at 12:51 PM, Tim Moody  wrote:
>>
>> Thanks, Anish for an excellent start at getting this down on paper (so to
>> speak).  I think this covers things pretty well and gives us the necessary
>> hooks on which to hang the details as we begin fleshing out solutions to
>> the requirements you documented.
>>
>>
>>
>> Under issues I added one point about rendering non-Roman character sets.
>>
>>
>>
>> *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com]
>> *On Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 12:15 PM
>> *To:* Community Support Volunteers -- who help respond to help AT
>> laptop.org
>> *Cc:* Unleash Kids!; server-devel; xsce-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* [UKids] Re: [support-gang] Taking OpenStreetMap Offline -
>> DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Okay, so I tried to encapsulate whatever we discussed into a design
>> document, which can hopefully serve as a base for reaching out to the OSM
>> community and to better organize our own thoughts.
>>
>>
>> https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit#
>> 
>>
>> Please fe