[datameet] Re: Help in Scraping Data
I am also glad sir it helps. On Thursday, May 28, 2020 at 9:33:04 PM UTC+5:30, Abhiroop Sarkar wrote: > > Hi Souvik, this is an extremely wonderful script. I just tried this and it > seemed to work great in all the different scenarios. Thanks a lot for the > help! > > On Thursday, May 28, 2020 at 6:18:49 PM UTC+5:30, Souvik Sankar Mitra > wrote: >> >> To download mvt of "Mumbai Existing Land Use" I have written this code. >> As the mvt files are stored according to zoom level . I run it from zoom >> level 10 to zoom level 21. This stores all mvt files w.r.t zoom level in >> different directory. Hope it helps. >> >> On Wednesday, May 27, 2020 at 6:32:38 PM UTC+5:30, Abhiroop Sarkar wrote: >>> >>> Hi, can someone help in extracting some data from >>> http://www.loginmumbai.org/map.html ? There is a lot of data on here >>> regarding Mumbai which can be helpful for further analysis. >>> >>> I have spent some time on this and from what I saw, all the layers are >>> displayed in 2 ways - >>> 1. Directly parsing TopoJSON through OpenLayers. >>> 2. Loading Vector Tiles (.mvt) through OpenLayers. >>> The smaller point and polygon layers are loaded directly whereas the >>> heavier polygon and polyline layers are loaded through the vector tiles. >>> >>> What I am having trouble with is extracting the vector tiles. For >>> example, I want the layer under "LandUse and Development Plans" > >>> "Mumbai Existing Land Use". Fetching the zoomed out .mvt file and >>> loading it in QGIS works but zooming in shows that the edges of the >>> polygons are very jagged. Zooming in and then fetching the .mvt file >>> produces perfectly clean geometry but only for a small region (only the >>> specific tile). >>> >>> Can someone help in getting such layers in a clean way? One of the ways >>> would be to automate the process of panning the zoomed in webmap to >>> gradually load the detailed .mvt files and save them automatically upon >>> loading, but I lack the coding expertise to implement this. Any and all >>> suggestions are welcome. >>> >>> Thank you. >>> >> -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/cd56a7d1-0de6-4dbb-a775-4dbe538025ce%40googlegroups.com.
[datameet] Re: Help in Scraping Data
Hi Souvik, this is an extremely wonderful script. I just tried this and it seemed to work great in all the different scenarios. Thanks a lot for the help! On Thursday, May 28, 2020 at 6:18:49 PM UTC+5:30, Souvik Sankar Mitra wrote: > > To download mvt of "Mumbai Existing Land Use" I have written this code. > As the mvt files are stored according to zoom level . I run it from zoom > level 10 to zoom level 21. This stores all mvt files w.r.t zoom level in > different directory. Hope it helps. > > On Wednesday, May 27, 2020 at 6:32:38 PM UTC+5:30, Abhiroop Sarkar wrote: >> >> Hi, can someone help in extracting some data from >> http://www.loginmumbai.org/map.html ? There is a lot of data on here >> regarding Mumbai which can be helpful for further analysis. >> >> I have spent some time on this and from what I saw, all the layers are >> displayed in 2 ways - >> 1. Directly parsing TopoJSON through OpenLayers. >> 2. Loading Vector Tiles (.mvt) through OpenLayers. >> The smaller point and polygon layers are loaded directly whereas the >> heavier polygon and polyline layers are loaded through the vector tiles. >> >> What I am having trouble with is extracting the vector tiles. For >> example, I want the layer under "LandUse and Development Plans" > >> "Mumbai Existing Land Use". Fetching the zoomed out .mvt file and >> loading it in QGIS works but zooming in shows that the edges of the >> polygons are very jagged. Zooming in and then fetching the .mvt file >> produces perfectly clean geometry but only for a small region (only the >> specific tile). >> >> Can someone help in getting such layers in a clean way? One of the ways >> would be to automate the process of panning the zoomed in webmap to >> gradually load the detailed .mvt files and save them automatically upon >> loading, but I lack the coding expertise to implement this. Any and all >> suggestions are welcome. >> >> Thank you. >> > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/8449bd7d-5304-460d-a908-c93622b09e51%40googlegroups.com.
[datameet] Re: Help in Scraping Data
To download mvt of "Mumbai Existing Land Use" I have written this code. As
the mvt files are stored according to zoom level . I run it from zoom level
10 to zoom level 21. This stores all mvt files w.r.t zoom level in
different directory. Hope it helps.
On Wednesday, May 27, 2020 at 6:32:38 PM UTC+5:30, Abhiroop Sarkar wrote:
>
> Hi, can someone help in extracting some data from
> http://www.loginmumbai.org/map.html ? There is a lot of data on here
> regarding Mumbai which can be helpful for further analysis.
>
> I have spent some time on this and from what I saw, all the layers are
> displayed in 2 ways -
> 1. Directly parsing TopoJSON through OpenLayers.
> 2. Loading Vector Tiles (.mvt) through OpenLayers.
> The smaller point and polygon layers are loaded directly whereas the
> heavier polygon and polyline layers are loaded through the vector tiles.
>
> What I am having trouble with is extracting the vector tiles. For example,
> I want the layer under "LandUse and Development Plans" > "Mumbai Existing
> Land Use". Fetching the zoomed out .mvt file and loading it in QGIS works
> but zooming in shows that the edges of the polygons are very jagged.
> Zooming in and then fetching the .mvt file produces perfectly clean
> geometry but only for a small region (only the specific tile).
>
> Can someone help in getting such layers in a clean way? One of the ways
> would be to automate the process of panning the zoomed in webmap to
> gradually load the detailed .mvt files and save them automatically upon
> loading, but I lack the coding expertise to implement this. Any and all
> suggestions are welcome.
>
> Thank you.
>
--
Datameet is a community of Data Science enthusiasts in India. Know more about
us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/datameet/1e85c040-21dd-49f6-a896-97421f2d77c5%40googlegroups.com.
import math
import numpy
import requests
import os
cur = os.getcwd()
def deg2num(lat_deg, lon_deg, zoom):
lat_rad = math.radians(lat_deg)
n = 2.0 ** zoom
xtile = int((lon_deg + 180.0) / 360.0 * n)
ytile = int((1.0 - math.asinh(math.tan(lat_rad)) / math.pi) / 2.0 * n)
return (xtile, ytile)
def unique(list1):
arr1=[]
# insert the list to the set
list_set = set(list1)
# convert the set to the list
unique_list = (list(list_set))
for x in unique_list:
arr1.append(x)
return arr1
#spatial extent of Mumbai
start = [19.308466, 72.785462]
end = [18.900295, 72.963473]
for b in range (10,22):
print ("Processing for zoom "+str(b))
arr=[]
for i in numpy.arange(end[0],start[0],0.01):
for j in numpy.arange(start[1],end[1],0.01):
arr.append(deg2num(i,j, b))
#print (arr)
arr2=[]
uni = unique(arr)
dir1 = 'zoom_'+str(b)
os.mkdir(dir1)
os.chdir(cur+'/'+dir1)
for a in uni:
url = 'http://119.81.20.14:8893/mvt/mcgm_landuse/'+str(b)+'/'+str(a[0])+'/'+str(a[1])+'.mvt'
r = requests.get(url, allow_redirects=True)
open(str(a[0])+'_'+str(a[1])+'.mvt', 'wb').write(r.content)
os.chdir(cur)
print ("Success")
