Re: [OSM-talk] Testing torrents for the planet dump

2020-02-10 Thread Christian Quest 

Le 09/02/2020 à 19:21, Yuri Astrakhan a écrit :
Christian, I would like to add torrent support to the download-osm 
tool [1].  While I could try to scrape
https://osm.cquest.org/torrents/ , I would obviously rather use the 
structured xml file (or if you could provide a JSON file, even better).



If you have a model for such a json, I can add it.



Proposed logic:
* get the catalog file (xml/json)
* download the latest torrent
* (TBD) possibly use some magical aria2c options to stop the download 
if it is not progressing and fallback to regular http, or possibly 
pick a slightly older torrent (?). Suggestions are welcome -- see full 
list of aria2c options [2]


[1] 
https://github.com/openmaptiles/openmaptiles-tools#multi-streamed-osm-data-downloader

[2] https://aria2.github.io/manual/en/html/aria2c.html

I've not tested aria2c, but I tested lftp which supports bittorrent, 
example:


lftp -c torrent 
https://osm.cquest.org/torrents/planet-latest.osm.pbf.torrent


After the download is done, it switches to a background process to share 
the file up to a ratio of 2:1.



Web seeds can be seen as fallback.

I don't know if lftp (or aria2c) supports web seeds, the lftp 
documentation I found about the torrent support is minimal.


aria2c support metalink files, which could be something to also look at: 
https://en.wikipedia.org/wiki/Metalink


--
Christian Quest - OpenStreetMap France


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Testing torrents for the planet dump

2020-02-09 Thread Yuri Astrakhan 
Christian, I would like to add torrent support to the download-osm tool
[1].  While I could try to scrape
https://osm.cquest.org/torrents/ , I would obviously rather use the
structured xml file (or if you could provide a JSON file, even better).

Proposed logic:
* get the catalog file (xml/json)
* download the latest torrent
* (TBD) possibly use some magical aria2c options to stop the download if it
is not progressing and fallback to regular http, or possibly pick a
slightly older torrent (?).  Suggestions are welcome -- see full list of
aria2c options [2]

[1]
https://github.com/openmaptiles/openmaptiles-tools#multi-streamed-osm-data-downloader
[2] https://aria2.github.io/manual/en/html/aria2c.html

On Sun, Feb 9, 2020 at 12:44 PM Christian Quest 
wrote:

>
> Le 09/02/2020 à 18:26, Maarten Deen a écrit :
> > On 2020-02-09 16:17, Christian Quest wrote:
> >> A couple of weeks ago, I've (re) started sharing planet dump files
> >> using Bittorrent to test an alternative way to distribute our planet
> >> dumps to reduce the bandwidth load on the OSMF servers.
> >>
> >> Here is a short summary after 2 weeks tests...
> >>
> >> The torrents are generated a few hours after the planet dump
> >> availability on planet.openstreetmap.org server (time needed to
> >> download the original file to generate the torrent).
> >
> > A question about updates of the torrent. The planet changes weekly.
> > Suppose I download the plannet using the torrent and then also seed
> > it, what happens when the next planet comes available? Do I need to
> > use a different torrent to download it again or will the one I have be
> > updated?
> Each planet file has its own torrent. A new planet file means using a
> new torrent to download it.
>
> To simplify downloading the lastest planet, you'll find a
> planet-latest.pbf.torrent similar to the planet-latest.pbf (it's a
> symlink to the last planet available).
>
> If you want to automate downloading new planet thru torrents, for
> example to seed them, there's a rss.xml file you can use. Many
> bittorrent clients support RSS to automate downloads.
>
> --
> Christian Quest - OpenStreetMap France
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Testing torrents for the planet dump

2020-02-09 Thread John Whelan 

My concern would be more the volume of data we have whirling round.

Smaller torrents of each country might be more useful and I'd have to 
get my head around picking up the new one each week.  I'm running 
qBittorrent if that is of any use.


Cheerio John

Christian Quest wrote on 2020-02-09 12:42 PM:


Le 09/02/2020 à 18:26, Maarten Deen a écrit :

On 2020-02-09 16:17, Christian Quest wrote:

A couple of weeks ago, I've (re) started sharing planet dump files
using Bittorrent to test an alternative way to distribute our planet
dumps to reduce the bandwidth load on the OSMF servers.

Here is a short summary after 2 weeks tests...

The torrents are generated a few hours after the planet dump
availability on planet.openstreetmap.org server (time needed to
download the original file to generate the torrent).


A question about updates of the torrent. The planet changes weekly. 
Suppose I download the plannet using the torrent and then also seed 
it, what happens when the next planet comes available? Do I need to 
use a different torrent to download it again or will the one I have 
be updated?
Each planet file has its own torrent. A new planet file means using a 
new torrent to download it.


To simplify downloading the lastest planet, you'll find a 
planet-latest.pbf.torrent similar to the planet-latest.pbf (it's a 
symlink to the last planet available).


If you want to automate downloading new planet thru torrents, for 
example to seed them, there's a rss.xml file you can use. Many 
bittorrent clients support RSS to automate downloads.




--
Sent from Postbox 
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Testing torrents for the planet dump

2020-02-09 Thread Christian Quest 


Le 09/02/2020 à 18:26, Maarten Deen a écrit :

On 2020-02-09 16:17, Christian Quest wrote:

A couple of weeks ago, I've (re) started sharing planet dump files
using Bittorrent to test an alternative way to distribute our planet
dumps to reduce the bandwidth load on the OSMF servers.

Here is a short summary after 2 weeks tests...

The torrents are generated a few hours after the planet dump
availability on planet.openstreetmap.org server (time needed to
download the original file to generate the torrent).


A question about updates of the torrent. The planet changes weekly. 
Suppose I download the plannet using the torrent and then also seed 
it, what happens when the next planet comes available? Do I need to 
use a different torrent to download it again or will the one I have be 
updated?
Each planet file has its own torrent. A new planet file means using a 
new torrent to download it.


To simplify downloading the lastest planet, you'll find a 
planet-latest.pbf.torrent similar to the planet-latest.pbf (it's a 
symlink to the last planet available).


If you want to automate downloading new planet thru torrents, for 
example to seed them, there's a rss.xml file you can use. Many 
bittorrent clients support RSS to automate downloads.


--
Christian Quest - OpenStreetMap France


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Testing torrents for the planet dump

2020-02-09 Thread Maarten Deen 

On 2020-02-09 16:17, Christian Quest wrote:

A couple of weeks ago, I've (re) started sharing planet dump files
using Bittorrent to test an alternative way to distribute our planet
dumps to reduce the bandwidth load on the OSMF servers.

Here is a short summary after 2 weeks tests...

The torrents are generated a few hours after the planet dump
availability on planet.openstreetmap.org server (time needed to
download the original file to generate the torrent).


A question about updates of the torrent. The planet changes weekly. 
Suppose I download the plannet using the torrent and then also seed it, 
what happens when the next planet comes available? Do I need to use a 
different torrent to download it again or will the one I have be 
updated?


Regards,
Maarten

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Testing torrents for the planet dump

2020-02-09 Thread Christian Quest 
A couple of weeks ago, I've (re) started sharing planet dump files using 
Bittorrent to test an alternative way to distribute our planet dumps to 
reduce the bandwidth load on the OSMF servers.


Here is a short summary after 2 weeks tests...

The torrents are generated a few hours after the planet dump 
availability on planet.openstreetmap.org server (time needed to download 
the original file to generate the torrent).


I'm seeding it from a couple of servers which allows to spread the dump 
file faster. Seeding is also done thru the original https link and main 
mirrors (this is the "web seed" feature supported by some bittorrent 
clients).


After the initial seed, downloading the planet file can be as short as 
30/40mn if you have a good downlink. After 3 days, there's currently 15 
seeds, allowing event faster or more balanced downloads (just did one in 
only 15mn).


If you have downloaded the planet dump using the torrent or plan to do 
so or want to test it, please share your results on the wiki discussion: 
https://wiki.openstreetmap.org/wiki/Talk:Planet.osm#Torrent


The torrents are located at https://osm.cquest.org/torrents/

--
Christian Quest - OpenStreetMap France


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk