Re: Scripted download of maps on Linux

[Poutnik Fornntp]

Attaching my Bash script for illustrative and inspirative purposes.

Note that I am not particularly good at Linux scripting, so it is surely 
possible to write in in more elegant or more powerful way. I have just 
written it recently, so updates are expected.


Dne 9. června 2019 17:33:45 Poutnik Fornntp  napsal:
Note also if one installs termux, a very good Linux terminal emulator, one 
can write wget or curl based Bash scripts for direct download to the final 
Android placement, optionally skipping downloads if no update is available, 
or if local files are not older than chosen threshold.


Note that it is by default sandboxed, must be explicitly configured to have 
access to Android storage.


I use it for BRouter RD5 file updates and Mapsforge compatible maps for 
LocusMap.(OpenAndroMaps globally, or osm.paws.cz for middle Europe.), using 
iteration overthe script arguments, where each argument is a code for 
particular downloadm defined withing tge script.


E.g. argument 0 means an update of my Brouter "homegrid" 5x5deg RD5 file to 
be downloaded to


~/storage/shared/Brouter/segments4


--
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osmand/16b3ce66ab0.2799.a291d67f9894f806060d35c996ca15e9%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
#!/bin/bash

#https://stackoverflow.com/questions/255898/how-to-iterate-over-arguments-in-a-bash-script
 
##  https://www.vionblog.com/linux-delete-files-older-than-x-days/
# find /path/to/files/ -type f -name '*.jpg' -mtime +30 -exec rm {} \;


webaddr=https://osm.paws.cz
#downloadfolder=~/Downloads
downloadfolder=~/storage/shared/Locus/mapsVector

websegments=http://brouter.de/brouter/segments4
segmentdownloadfolder=~/storage/shared/Brouter/segments4

function getfile0 {
cd $downloadfolder
wget -N  $webaddr/$1
}

function getfile {
cd $downloadfolder
maxdayage=40
find $downloadfolder/  -type f -name $1 -mtime +$maxdayage -exec wget -N 
$webaddr/$1 \;

}



function getrd5file0 {
# %1 = lon  E5,E10,   %2=lat
cd $segmentdownloadfolder
wget -N --limit-rate=10M $websegments/$1_$2.rd5
}

function getrd5file {
cd $segmentdownloadfolder
rd5file=$1_$2.rd5
maxdayage=20
find $segmentdownloadfolder/  -type f -name $rd5file -mtime +$maxdayage -exec 
wget -N --limit-rate=10M $websegments/$rd5file \;
}

for var in "$@"
do

#   https://www.shellscript.sh/case.html

case $var in
0)
getrd5file E15 N45
;;
1)
getrd5file E10 N45
getrd5file E10 N50
getrd5file E15 N50
;;
2)
getrd5file E10 N40
getrd5file E15 N40
getrd5file E20 N40 
getrd5file E20 N45 
getrd5file E20 N50 
;;
3)
getrd5file E0 N40 
getrd5file E0 N45 
getrd5file E0 N50
getrd5file E5 N40 
getrd5file E5 N45 
getrd5file E5 N50
;;
4)
getrd5file E20 N50 
getrd5file E20 N55
getrd5file E25 N50
getrd5file E25 N55 
;;
cz)
getfile czech_republic_gccz.map
;;
sk)
getfile slovakia_gccz.map
;;
at)
getfile austria_gccz.map
;;
ato)
wget -N http://download.openandromaps.org/maps/europe/Austria.zip
unzip Austria.zip -o
rm Austria.zip
;;
at4)
wget -N http://download.openandromaps.org/mapsV4/europe/Austria.zip
unzip Austria.zip -o
rm Austria.zip
;;
sa)
getfile sachsen_gccz.map
;;
ba)
getfile bayern_gccz.map
;;
ine)
getfile nord_est_gccz.map
;;
inw)
getfile nord_ovest_gccz.map
;;
cr)
getfile croatia_gccz.map
;;
sl)
getfile slovenia_gccz.map
;;
hu)
getfile hungary_gccz.map
;;
sw)
getfile switzerland_gccz.map
;;
uk)
getfile ukraine_gccz.map
;;
pl)
getfile poland_gccz.map.zip
unzip poland_gccz.map.zip -o
rm poland_gccz.map.zip
;;
*)  echo "Unknown code"
;;
esac

done

Re: Scripted download of maps on Linux

[Poutnik Fornntp]

Note also if one installs termux, a very good Linux terminal emulator, one 
can write wget or curl based Bash scripts for direct download to the final 
Android placement, optionally skipping downloads if no update is available, 
or if local files are not older than chosen threshold.



Note that it is by default sandboxed, must be explicitly configured to have 
access to Android storage.



I use it for BRouter RD5 file updates and Mapsforge compatible maps for 
LocusMap.(OpenAndroMaps globally, or osm.paws.cz for middle Europe.), using 
iteration overthe script arguments, where each argument is a code for 
particular downloadm defined withing tge script.



E.g. argument 0 means an update of my Brouter "homegrid" 5x5deg RD5 file to 
be downloaded to



~/storage/shared/Brouter/segments4





--
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osmand/16b3cdfb7d8.2799.a291d67f9894f806060d35c996ca15e9%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

['Sabine Mayer' via Osmand]

Hi, 
thats a cute script and does look much better than one I currently use!

But I also to download the srtm-files and the wiki files but the links do 
no longer work.
srtm used to be at http://download.osmand.net/srtm-countries/
and wiki at http://download.osmand.net/wiki/

do you know where these files were moved to?
cheers,
sabby

-- 
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osmand/106a152e-1c0c-4712-a74f-e70def1c41e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[CP]

You cannot cache HTTPS traffic. That goes against the design of HTTPS.
I you want to distribute maps locally and automatically , your best 
option is some kind of p2p setup (syncthing/torrent)


CP




Op 15-03-18 om 00:16 schreef Rob:
For some time I've been thinking to do something like this but never 
got round to it. I was planning to set up a caching proxy and 
configure my home router to redirect only traffic for the Osmand map 
server there. Similar idea but less overhead - just download the maps 
on one device and they are automatically cached on the local network 
for subsequent devices. I hope this would also be a cross platform 
solution (iOS/Android) but I'm not sure whether the 2 apps use the 
same map files.


Curious to know if anyone out there has already tried this setup?

R

On Sunday, 11 March 2018 17:01:54 UTC, We Ka wrote:

Hello,

I am using Osmand+ (paid version) on several devices and I think
it's highly inefficient to download each map on each device,
unnecessary increasing data usage. Therefore, I'd like to download
updated maps to my pc and copy them onto the sdcards afterwards. I
know I can do "wget

http://download.osmand.net/download.php?standard=yes&file=Germany_baden-wuerttemberg_europe_2.obf.zip

"
to get one map, but there are two problems with that:

1. Automatic file name completion doesn't work, so I can't use a
command like "wget
http://download.osmand.net/download.php?standard=yes&file=Germany_*.zip
",
I have to use a wget for each single file I want to download.

2. After downloading, the files are stored as
"download.php?standard=yes&file=Germany_bremen_europe_2.obf.zip"
to the local hard disk, and I have to remove the part in front of
"Germany".

How can I solve these two problems? Is there any other way of
limiting the download to a minimum and still get the maps onto
every device?

--
You received this message because you are subscribed to the Google 
Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to osmand+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[Rob]

For some time I've been thinking to do something like this but never got 
round to it. I was planning to set up a caching proxy and configure my home 
router to redirect only traffic for the Osmand map server there. Similar 
idea but less overhead - just download the maps on one device and they are 
automatically cached on the local network for subsequent devices. I hope 
this would also be a cross platform solution (iOS/Android) but I'm not sure 
whether the 2 apps use the same map files. 

Curious to know if anyone out there has already tried this setup?

R

On Sunday, 11 March 2018 17:01:54 UTC, We Ka wrote:
>
> Hello,
>
> I am using Osmand+ (paid version) on several devices and I think it's 
> highly inefficient to download each map on each device, unnecessary 
> increasing data usage. Therefore, I'd like to download updated maps to my 
> pc and copy them onto the sdcards afterwards. I know I can do "wget 
> http://download.osmand.net/download.php?standard=yes&file=Germany_baden-wuerttemberg_europe_2.obf.zip";
>  
> to get one map, but there are two problems with that:
>
> 1. Automatic file name completion doesn't work, so I can't use a command 
> like "wget 
> http://download.osmand.net/download.php?standard=yes&file=Germany_*.zip";, 
> I have to use a wget for each single file I want to download.
>
> 2. After downloading, the files are stored as 
> "download.php?standard=yes&file=Germany_bremen_europe_2.obf.zip" to the 
> local hard disk, and I have to remove the part in front of "Germany".
>
> How can I solve these two problems? Is there any other way of limiting the 
> download to a minimum and still get the maps onto every device?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[CP]

Op 13-03-18 om 09:27 schreef We Ka:

Hey, that is a pretty cute script. Thank you very much! :)


You're welcome.


Yet, there are a few things that needed to be corrected to make it work:

1. the URL's hostname needs to be "download", not "dl3", in order to 
be resolved. (I guess there's some load balancing happening there.)


Agreed. In my reality download works just fine.

2. the line containing the grep command needs to end with a "\ ". 
(That's two characters: a backslash followed by a space, but without 
quotes around them.)


No, that is not needed. If you copy this to a bash script it works. I 
think you've fallen victim to a local issue of command line and/or shell


3. curl doesn't seem to be able to handle the actual URL containing 
the "&" (ampersand). I had to replace curl by wget in order to make 
the download work:

|
           wget "${BASE_URL}${MAP}"-O "${STOR}/${MAP}"
|



This is probably related to the curl version. Depending on your 
distribution you might get a different curl.

Mine is:

koos@zwaluw:~> curl --version
curl 7.37.0 (x86_64-suse-linux-gnu) libcurl/7.37.0 OpenSSL/1.0.2j 
zlib/1.2.8 libidn/1.28 libssh2/1.4.3


Apart from that: very handy! Now I can get the list of files every 
week or so, compare it to the version from the last check, and if 
there's a difference, I can download the maps and distribute them to 
all of my Android devices inside my home network using "OwnCloud" and 
"foldersync".


Yup :-)

--
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[Poutnik]

I see this batch version uses an obsolete wget version from Gnuwin32
project.
Windows users of cygwin can easily switch to the cygwin wget version.

Dne 13/03/2018 v 09:36 Poutnik napsal(a):
> As I once wrote and occasionally use
> a cygwin/wget based batch for maps download in Windows,
> I do not see a big deal in writing a script for Linux.
>
> As inspiration ( older, but should work, even if not checked for a while )
> Github : /Brouter-profiles/master/OSMANd_wget_maps
> 
> ( will be displayed as the CMD content and can be directly saved as
> CMD file )

-- 
Poutnik ( The Wanderer )

My Brouter profiles 
https://github.com/poutnikl/Brouter-profiles/wiki

-- 
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[Poutnik]

As I once wrote and occasionally use
a cygwin/wget based batch for maps download in Windows,
I do not see a big deal in writing a script for Linux.

As inspiration ( older, but should work, even if not checked for a while )
Github : /Brouter-profiles/master/OSMANd_wget_maps

( will be displayed as the CMD content and can be directly saved as CMD
file )

Dne 11/03/2018 v 18:01 We Ka napsal(a):
> Hello,
>
> I am using Osmand+ (paid version) on several devices and I think it's
> highly inefficient to download each map on each device, unnecessary
> increasing data usage. Therefore, I'd like to download updated maps to
> my pc and copy them onto the sdcards afterwards. I know I can do "wget
> http://download.osmand.net/download.php?standard=yes&file=Germany_baden-wuerttemberg_europe_2.obf.zip";
> to get one map, but there are two problems with that:
>
> 1. Automatic file name completion doesn't work, so I can't use a
> command like "wget
> http://download.osmand.net/download.php?standard=yes&file=Germany_*.zip";,
> I have to use a wget for each single file I want to download.
>
> 2. After downloading, the files are stored as
> "download.php?standard=yes&file=Germany_bremen_europe_2.obf.zip" to
> the local hard disk, and I have to remove the part in front of "Germany".
>
> How can I solve these two problems? Is there any other way of limiting
> the download to a minimum and still get the maps onto every device?

-- 
Poutnik ( The Wanderer )

My Brouter profiles 
https://github.com/poutnikl/Brouter-profiles/wiki

-- 
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[We Ka]

Hey, that is a pretty cute script. Thank you very much! :)

Yet, there are a few things that needed to be corrected to make it work: 

1. the URL's hostname needs to be "download", not "dl3", in order to be 
resolved. (I guess there's some load balancing happening there.)

2. the line containing the grep command needs to end with a "\ ". (That's 
two characters: a backslash followed by a space, but without quotes around 
them.)

3. curl doesn't seem to be able to handle the actual URL containing the "&" 
(ampersand). I had to replace curl by wget in order to make the download 
work:
   wget "${BASE_URL}${MAP}" -O "${STOR}/${MAP}"

Apart from that: very handy! Now I can get the list of files every week or 
so, compare it to the version from the last check, and if there's a 
difference, I can download the maps and distribute them to all of my 
Android devices inside my home network using "OwnCloud" and "foldersync". 

Thank you once more for the script!

-- 
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scripted download of maps on Linux

[CP]

BASE_URL='http://dl3.osmand.net/download.php?standard=yes&file='
STOR=/mnt/vol1/algemeen/OsmAnd/maps
COUNTRIES="Italy Germany Austria Netherlands Switzerland Us_new-york 
Us_south-carolina"


curl http://download.osmand.net/list.php | sed -e 's//\'$'\n/g' 
> osm_maps.html

for COUNTRY in ${COUNTRIES}; do
    grep ${COUNTRY} osm_maps.html |sed -e 's/^.*file=//' | sed -e 
's/">.*$//' |

    while read MAP; do
    echo "${MAP}"
    ( curl ${BASE_URL}${MAP} > ${STOR}/${MAP} 2>/dev/null )
    if [[ ${MAP} =~ .zip$ ]]; then
    unzip -o -d ${STOR} ${STOR}/${MAP} && rm ${STOR}/${MAP}
    fi;
    done
done

Have fun!

CP


Op 11-03-18 om 18:01 schreef We Ka:

Hello,

I am using Osmand+ (paid version) on several devices and I think it's 
highly inefficient to download each map on each device, unnecessary 
increasing data usage. Therefore, I'd like to download updated maps to 
my pc and copy them onto the sdcards afterwards. I know I can do "wget 
http://download.osmand.net/download.php?standard=yes&file=Germany_baden-wuerttemberg_europe_2.obf.zip"; 
to get one map, but there are two problems with that:


1. Automatic file name completion doesn't work, so I can't use a 
command like "wget 
http://download.osmand.net/download.php?standard=yes&file=Germany_*.zip";, 
I have to use a wget for each single file I want to download.


2. After downloading, the files are stored as 
"download.php?standard=yes&file=Germany_bremen_europe_2.obf.zip" to 
the local hard disk, and I have to remove the part in front of "Germany".


How can I solve these two problems? Is there any other way of limiting 
the download to a minimum and still get the maps onto every device?

--
You received this message because you are subscribed to the Google 
Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to osmand+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osmand+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.