Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[yvecai]

A bit of apache config to contribute.
At OsmAnd, we started building a tile server, however it is not very 
powerfull, and is pre-rendered up to zoom 13 only. Thus we are making 
some attempts to have a load balancer on another server, it look like this:



ServerAdmin webmaster@localhost
ServerName loadbalancer.osmand.net

 Order allow,deny
 Allow from all


 BalancerMember http://tileserver.osmand.net status=+I
 BalancerMember http://tile.openstreetmap.org status=+H

ProxyPass / balancer://hotcluster/

LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{X-Forwarded-Server}i\" 
\"%{Referer}i\" \"%{User-Agent}i\" duration:%T/%D io:%$

CustomLog ${APACHE_LOG_DIR}/access.log lb



When a tile is requested to loadbalancer.osmand.net, it is proxied to 
tileserver.osmand.net.
On this tile server, mod_tile will return a 404 if the requested tile is 
not available after 5 seconds. After this 404, then 
loadbalancer.osmand.net request the tile to tile.openstreetmap.org until 
tileserver.osmand.net is back.


Any comment, flaws detected or hints highly appreciated.

Yves


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Peter Körner]

Am 30.12.2011 19:17, schrieb Stefan de Konink:

I do not have yet investigated what the options are of CoralCDN, but
the main problem is to get the *alternative used*. I guess switching
to a system that is partly in our own hands results in a higher
quality, thus we must have ability to invalidate data quickly.


Tirex is able to be used as cache against a wms. It should be able to 
work as a cache against another tile server, supports expiring and 
rerendering rules and stores content as metatiles.


I guess a cache could be implemented as tirex renderer.

Peter


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Stefan de Konink]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Op 03-01-12 09:21, Philipp Borgers schreef:
> We can provide rack space and bandwidth for at least one server
> here in Berlin. We also have some servers we can contribute to the
> project but they are not that powerful.

I hope you can put your offer on the Wiki;


A real real CDN just requires local bandwidth, a harddrive and a
static webserver. It is therefore interesting to offer 'local' end
nodes there were traffic is free due to peering, for example.


> The major problem are low zoom level tiles which are rendered
> adhoc. We can't solve this problem with tile caches as far as I
> know. Should every node in the CDN be able to render tiles?

No, the CDN nodes will al be static. Some intelligent behaviour is
envisioned for tiles that are not rendered.


> Is there some kind of global render cluster where caching nodes
> connect to?

The other way around, nodes receive updates, push based.


> How do we keep the data/database for rendering in sync?

Like every other tile server does. And maintaining central information
on last updates per participant in the CDN.


> Do we offer different tile styles?

First things first: the default map style.


> Is CDN usage free of charge?

Yes, obviously. I envision that something can be done using the
attribution text to present a status/powered by notification.


Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAk8DRcwACgkQYH1+F2Rqwn1gCgCgi7xMqCuY3xbvXTWjzEwKamHn
LAAAnjZpAw0qFtQUsuuNKu9pRZ0Ql+mz
=j+uU
-END PGP SIGNATURE-

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Philipp Borgers]

We can provide rack space and bandwidth for at least one server here in
Berlin. We also have some servers we can contribute to the project but
they are not that powerful.

The major problem are low zoom level tiles which are rendered adhoc. We
can't solve this problem with tile caches as far as I know. Should every
node in the CDN be able to render tiles? Is there some kind of global
render cluster where caching nodes connect to? How do we keep the
data/database for rendering in sync?

Do we offer different tile styles?

Is CDN usage free of charge?

On Fri, 2011-12-30 at 18:10 +0100, Stefan de Konink wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> The basic outline;
> 
> - - People seem to love using tile.openstreetmap.org for any of their apps
> - - Apps get blocked for various reasons, contributing back is difficult
> - - OpenStreetMap wants to be the leading provider of such data
> 
> 
> I would propose a small working group to outline what should happen to
> facilitate the creation of a Content Delivery Network (CDN) serving
> the basic tiles, and allow easy contribution this network. This should
> produce a prototype of at least 3 servers.
> 
> 
> Questions such as: quality, update frequency, traffic shaping,
> geographical balancing and high availability could be part of this
> working group.
> 

We should define in addition some requirements for servers participating
in the network like:

* Availability
 * Server
 * Network
* min. Bandwidth
* min. RAM/DISK space
* max. access time
* root-access for admins of the CDN

> 
> I would like to invite anyone to participate, especially:
>  - people that already have their own tileservers running, and/or
>are currently balancing traffic;
>  - business folks: what could be a motivation and what can be a
>cutback in for example attribution,
>  - users of for example openlayers, etc. what kind of caching can
>be applied, and if this should be configurable client side
> 
> 
> Participation can announced in private or on list.
> 
> 
> Stefan
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.18 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEAREKAAYFAk798H0ACgkQYH1+F2Rqwn2I9ACeJV+vFfBvI4CaTET0BqjuuX1s
> G78AnRv6+Qh7MkIrN1DQ6fQ34p1j0uuS
> =+WNS
> -END PGP SIGNATURE-
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev



signature.asc
Description: This is a digitally signed message part
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Mike Dupont]

John smith has also already done experiments with hosting on
archive.org he has tools for splitting up the tiles etc.
mike

On Mon, Jan 2, 2012 at 11:35 PM, yvecai  wrote:
> What about tile update? Can they handle a tile server or just caching?
> I guess they could prefer metatiles to tiles.
>
> Yves
>
>
> On 02. 01. 12 23:16, Stefan de Konink wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>>
>> Archive.org replied positively; update on the wiki.
>>
>> 
>>
>>
>> Stefan
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v2.0.18 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEAREKAAYFAk8CLMQACgkQYH1+F2Rqwn0PkwCggIDTSQtuqNYVUFo/8HoQsbgA
>> dvIAni4EbJnciIOmiKRcw3qfiklWjX8f
>> =ubKr
>> -END PGP SIGNATURE-
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/dev
>>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev



-- 
James Michael DuPont
Member of Free Libre Open Source Software Kosova http://flossk.org

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[yvecai]

What about tile update? Can they handle a tile server or just caching?
I guess they could prefer metatiles to tiles.

Yves

On 02. 01. 12 23:16, Stefan de Konink wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Archive.org replied positively; update on the wiki.




Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAk8CLMQACgkQYH1+F2Rqwn0PkwCggIDTSQtuqNYVUFo/8HoQsbgA
dvIAni4EbJnciIOmiKRcw3qfiklWjX8f
=ubKr
-END PGP SIGNATURE-

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev




___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Stefan de Konink]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Archive.org replied positively; update on the wiki.




Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAk8CLMQACgkQYH1+F2Rqwn0PkwCggIDTSQtuqNYVUFo/8HoQsbgA
dvIAni4EbJnciIOmiKRcw3qfiklWjX8f
=ubKr
-END PGP SIGNATURE-

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Stefan de Konink]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

I have just summerized the e-mail by Mike, and created a wiki page the
follow up the status:



Op 30-12-11 18:23, Lynn W. Deffenbaugh (Mr) schreef:
> I would love to monitor such discussion and contribute if I think
> I have something to offer.  Will such discussion be here on the 
> [OSM-dev} or elsewhere?

I don't want to induce a lot of offtopic and negatively. The 30 days
in the subject is something that I would like to realise, not talk about.


> Theoretically the CoralCDN could be used for distributing OSM
> tiles by simply suffixing the domain with ".nyud.net", but I have
> not tried this yet.  There would be little to no intelligence
> brought to bear on how long individual tiles would be cached vs
> flushed and/or whether mixed-revision tiles would be present in
> and delivered from the cache, not to mention that I don't think
> the CoralCDN passes through the original User-Agent either.

I do not have yet investigated what the options are of CoralCDN, but
the main problem is to get the *alternative used*. I guess switching
to a system that is partly in our own hands results in a higher
quality, thus we must have ability to invalidate data quickly.

If that is covered in CoralCDN the only thing remaining is finding a
good solution for the 'hostname' issue.


Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAk7+AE4ACgkQYH1+F2Rqwn3V1ACfd9wODnhcav/lC0jnaK816/AX
YAwAnig+5IuGoiuxciB32FHl3ulnMvjk
=pKVk
-END PGP SIGNATURE-

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Lynn W. Deffenbaugh (Mr)]

I would love to monitor such discussion and contribute if I think I have 
something to offer.  Will such discussion be here on the [OSM-dev} or 
elsewhere?


I currently use the freely available CoralCDN (http://www.coralcdn.org/) 
to distribute updates to my APRSISCE/32 APRS client software 
(http://aprsisce.wikidot.com/), but have mixed results (like only about 
50% success, but every cached delivery saves my original bandwidth).  
Thankfully I built in a direct backup link that picks up the slack from 
the failures.


Theoretically the CoralCDN could be used for distributing OSM tiles by 
simply suffixing the domain with ".nyud.net", but I have not tried this 
yet.  There would be little to no intelligence brought to bear on how 
long individual tiles would be cached vs flushed and/or whether 
mixed-revision tiles would be present in and delivered from the cache, 
not to mention that I don't think the CoralCDN passes through the 
original User-Agent either.


Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32

On 12/30/2011 12:10 PM, Stefan de Konink wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

The basic outline;

- - People seem to love using tile.openstreetmap.org for any of their apps
- - Apps get blocked for various reasons, contributing back is difficult
- - OpenStreetMap wants to be the leading provider of such data


I would propose a small working group to outline what should happen to
facilitate the creation of a Content Delivery Network (CDN) serving
the basic tiles, and allow easy contribution this network. This should
produce a prototype of at least 3 servers.


Questions such as: quality, update frequency, traffic shaping,
geographical balancing and high availability could be part of this
working group.


I would like to invite anyone to participate, especially:
  - people that already have their own tileservers running, and/or
are currently balancing traffic;
  - business folks: what could be a motivation and what can be a
cutback in for example attribution,
  - users of for example openlayers, etc. what kind of caching can
be applied, and if this should be configurable client side


Participation can announced in private or on list.


Stefan
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAk798H0ACgkQYH1+F2Rqwn2I9ACeJV+vFfBvI4CaTET0BqjuuX1s
G78AnRv6+Qh7MkIrN1DQ6fQ34p1j0uuS
=+WNS
-END PGP SIGNATURE-

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev




___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] What can we do to get a tile.openstreetmap.org contributable CDN within a month?

[Mike Dupont]

I have mentioned the fact many times that archive.org has allowed for
tile hosting,
i...@archive.org is the address for asking permission.
mike

On Fri, Dec 30, 2011 at 6:10 PM, Stefan de Konink  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> The basic outline;
>
> - - People seem to love using tile.openstreetmap.org for any of their apps
> - - Apps get blocked for various reasons, contributing back is difficult
> - - OpenStreetMap wants to be the leading provider of such data
>
>
> I would propose a small working group to outline what should happen to
> facilitate the creation of a Content Delivery Network (CDN) serving
> the basic tiles, and allow easy contribution this network. This should
> produce a prototype of at least 3 servers.
>
>
> Questions such as: quality, update frequency, traffic shaping,
> geographical balancing and high availability could be part of this
> working group.
>
>
> I would like to invite anyone to participate, especially:
>  - people that already have their own tileservers running, and/or
>   are currently balancing traffic;
>  - business folks: what could be a motivation and what can be a
>   cutback in for example attribution,
>  - users of for example openlayers, etc. what kind of caching can
>   be applied, and if this should be configurable client side
>
>
> Participation can announced in private or on list.
>
>
> Stefan
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.18 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEAREKAAYFAk798H0ACgkQYH1+F2Rqwn2I9ACeJV+vFfBvI4CaTET0BqjuuX1s
> G78AnRv6+Qh7MkIrN1DQ6fQ34p1j0uuS
> =+WNS
> -END PGP SIGNATURE-
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev



-- 
James Michael DuPont
Member of Free Libre Open Source Software Kosova http://flossk.org

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev