Re: [datameet] Re: Location of all protected heritage monuments in India

[Nikhil VJ]

Hi All,

Devdatta posted on another thread a python program he made for converting
even lines and polygons to .geojson format and it extracts the metadata too.
Link to conversation:
https://groups.google.com/d/msg/datameet/XTF4_P-nxU8/LS0_d85YBwAJ
Program link: https://github.com/devdattaT/GeoRSSConverter

Coincidentally I was also making something along the same lines over the
weekend and didn't notice the update there.
So here's another path up the mountain:
https://github.com/answerquest/georss2geojson-metadata-rescue

We've worked with different libraries, and I'm posting mine too for these
reasons:
- I've used a 'feedparser' module that does most of the grunt work of
converting the XML to geojson formatted objects. Which gave me time to:
- Used geojson and geojson_utils packages that can be taken forward for
programmatically doing GIS-type operations like merging separated parts
into multi-polygon (which is something mentioned about Bhuvan's data),
computing distance and area, detecting intersections etc
- Parsed the numbers in the metadata even if they're like: "-3.34e-4". So
the output .geojson shapefile has numeric fields if the metadata was
carrying it, one can directly use them to render choropleth etc. See a bad
example .
- Did a "right hand rule" rectification that was needed to produce valid
GeoJSON that checks out on geojsonlint.com . (not critical but some places
still need it)
- Save an additional CSV where the metadata is laid out in a neat table,
and if it's a point locations file, then the csv will have lat-long columns
and can be used directly for mapping, like I did with the asi-monuments
data previously .

I've made it in a python 3 Jupyter Notebook ; We could
make this into a user-friendly website or a EXE program that runs on
windows, but that will take some work. Contact if interested.


*Extra: *
1. Here's a site that makes embeddable choropleth maps without coding
needed, and can import the numbers from outside: https://app.datawrapper.de
(yes, QGIS can do it too, but single-task-focus brings user-friendliness)
2. And here are two sites that convert shapefiles between various formats:
http://ogre.adc4gis.com, https://geoconverter.hsr.ch/ (they don't rescue
the trapped metadata of course, that's peculiar to Bhuvan)


--
Cheers,
Nikhil VJ
+91-966-583-1250
Pune, India
http://nikhilvj.co.in


On Wed, Oct 17, 2018 at 10:57 AM Sashikumar N 
wrote:

> HI Nikhil,
> Cool and neat solution, Thankyou. just a quick question, what if the WMS
> layer is a line or polygon, what has to be changed in the python script?
>
> regards
> sashi
>
> On Wednesday, October 17, 2018 at 10:05:34 AM UTC+5:30, Nikhil VJ wrote:
>>
>> Here you go,
>>
>> https://github.com/answerquest/asi-monuments-xml2csv
>>
>> It's in a CSV now and you can drag-drop it on geojson.io for preview,
>> and convert it to other formats from there.
>>
>> All metadata encountered in the xml is saved in respective columns.
>>
>> The python script used is there too, in a jupyter notebook (.ipynb) file.
>> Github will display it nicely.
>>
>> -Nikhil VJ
>> Pune, India
>>
>>
>> On Tuesday, October 16, 2018 at 2:42:43 PM UTC+5:30, Arun Ganesh wrote:
>>>
>>> Found the Bhuvan geoserver listing which has the asi layers:
>>> http://bhuvan5.nrsc.gov.in/bhuvan/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage
>>>
>>> They have disabled WFS so its not possible to download the shapefiles,
>>> but it looks the WMS GeoRSS output gives the vectors in an xml format.
>>>
>>> Monument locations:
>>> http://bhuvan5.nrsc.gov.in/bhuvan/asi/wms?service=WMS=1.1.0=GetMap=asi:monuments2==68.963,8.076,96.005,32.976=512=471=EPSG:4326=application%2Frss%2Bxml
>>>
>>> The file opens great in Qgis, but the feature properties are formatted
>>> as an HTML list. If this can be cleaned up, we can have the exact
>>> shapefiles.
>>>
>>> On Tue, Oct 16, 2018 at 12:59 PM Sashikumar N 
>>> wrote:
>>>
 Scrapping WMS layer is not possible, (i don't know if there is a way to
 do it). But if your interest is only few entities, you can use 'Tools' ->
 'Draw'  to mark/draw and download them as shapefile from the Bhuvan itself.

 On Monday, October 15, 2018 at 1:31:03 PM UTC+5:30, Arun Ganesh wrote:
>
>
>
> On Mon, Oct 15, 2018 at 11:50 AM Sashikumar N 
> wrote:
>
>> Bhuvan has a WMS layer of all ASI monuments, please check
>>
>
> Thanks, found it here:
> https://bhuvan-app1.nrsc.gov.in/culture_monuments/
>
> Its fantastic to browse around and should be shared more widely. Is
> there a way to get the features as vectors?
>
 --
 Datameet is a community of Data Science enthusiasts in India. Know more
 about us by visiting http://datameet.org
 ---
 You received this message because you are subscribed to the Google
 Groups "datameet" group.
 To 

[datameet] Re: Crowdsourcing Mobile Application by Survey of India

[Naveen Francis]

I couldn't find the apk. 
On the facebook page of SOI, mobile app screenshots are available, poster 
on August 15,2018 

On Monday, 29 October 2018 09:16:21 UTC+5:30, Naveen Francis wrote:
>
> Hello,
>
>
> http://www.surveyofindia.gov.in/pages/display/257-%E0%A4%B8%E0%A4%B9%E0%A4%AF%E0%A5%8B%E0%A4%97--mobile-app-by-soi
> http://www.surveyofindia.gov.in/files/Sahyog%20Mobile%20App%20by%20SoI.pdf
>
> --
> naveenpf
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[datameet] Crowdsourcing Mobile Application by Survey of India

[Naveen Francis]

Hello,

http://www.surveyofindia.gov.in/pages/display/257-%E0%A4%B8%E0%A4%B9%E0%A4%AF%E0%A5%8B%E0%A4%97--mobile-app-by-soi
http://www.surveyofindia.gov.in/files/Sahyog%20Mobile%20App%20by%20SoI.pdf

--
naveenpf

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[datameet] Re: Official Survey Of India - India Boundary

[Justin]

I'm 90% certain I know the source of the soi boundaries. I'll check in the 
morning. 

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [datameet] Re: India Shape Files

[Devdatta Tengshe]

Hi Arun,

I wrote a simple script for converting this GeoRSS to GeoJSON, along with
the attributes.
https://github.com/devdattaT/GeoRSSConverter

Do check it out and let me know.

Regards,
Devdatta


On Sun, Oct 28, 2018 at 11:58 AM Arun Ganesh  wrote:

> The various village/taluk shapefile layers are available in the Bhuvan
> GeoServer layer listing:
>
>
> http://bhuvannuis.nrsc.gov.in/bhuvan/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage
>
> Search for 'vill' for village boundaries by state. Search for 'taluk' for
> all taluk boundaries. Note that the page gives a timeout error if you take
> more than a few seconds and you need to start over. Since WFS export is
> disabled, you can use the GeoRSS option to download the shape.
>
> The one issue is that the attributes are encoded as an html string and
> needs to be parsed out. Nikhil has a great python script to do this:
> https://github.com/answerquest/asi-monuments-xml2csv/blob/master/asi-monuments2.ipynb
> but needs to be customized for each layer.
>
>
>
> On Fri, May 11, 2018 at 1:48 PM Sharad Lele  wrote:
>
>> Hi Thejesh:
>>
>> Did you ever manage to convert the data into a polygon file? I could not
>> follow the technical intricacies of what you were discussing with Palash,
>> but certainly the original json file posted by him did not contain the
>> polygons and could not be imported into QGIS. Do you now have one which can
>> be? And is it good quality?
>>
>> Sharad
>>
>> On Friday, January 20, 2017 at 6:24:51 PM UTC+5:30, Thejesh GN wrote:
>>>
>>> Imported into mongo to do some initial analysis[1].
>>>
>>> Will do some more analysis.
>>>
>>>
>>> > db.states.count()
>>> 36
>>> > db.districts.count()
>>> 672
>>> > db.taluks.count()
>>> 5960
>>> > db.villages.count()
>>> 522291
>>>
>>>
>>>
>>>
>>> [1]  https://gist.github.com/thejeshgn/ea0074077b6fcd6c0108b2a4380fc354
>>>
>>> Thej
>>> --
>>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>>> http://thejeshgn.com
>>> GPG ID :  0xBFFC8DD3C06DD6B0
>>>
>>> On 20 January 2017 at 17:03, Palash Kulshrestha 
>>> wrote:
>>>
 Hi Thejesh
 Thanks for the heads up, I wasn't aware of the underlying architecture
 which is used in geo systems. So wasn't able to understand the layers data.
 And now I am able to understand how they come up with polygon on the map.
 Other wise I was thinking that they somehow came up random shapes using min
 max x,y.

 I'm looking into the new api that you have provided and see what I can
 come up with. I ll update. Any advice?

 On Friday, January 20, 2017 at 4:54:47 PM UTC+5:30, Thejesh GN wrote:
>
> @Palash Kulshrestha
> Let me know if you downloaded the shapes too.
>
>
> The accuracy isn't bad for a national level project
>
> https://gist.github.com/thejeshgn/2b50ce7fe45287152f55941bd8d76869
>
>
>
>
>
> Thej
> --
> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
> GPG ID :  0xBFFC8DD3C06DD6B0
>
> On 20 January 2017 at 16:47, Thejesh GN  wrote:
>
>> Ah
>>
>>
>> https://ncog.gov.in/garvgis/admin/getTalukaWiseVillageBoundary?districtId=572=05545=613256
>>
>> Gives
>>
>> {"gid":438129,"vilNameSoi":"KAMMASANDRA","stateId":"29","distId":"572","subdistId":"05545","villageId":"613256","villageName":"Kammasandra","minx":77.68262864449315,"miny":12.824103451365033,"maxx":77.69996896356992,"maxy":12.841455112691222}
>>
>>
>> Then you get the actual layer using
>>
>> https://ncog.gov.in/garvgis/admin/ows?service=WFS=1.0.0=GetFeature=cite:Village2014=the_geom,VIL_2011,VIL_NAME11_FILTER=VIL_2011=%27613256%27=50=json
>>
>> {"type":"FeatureCollection","totalFeatures":1,"features":[{"type":"Feature","id":"Village2014.438129","geometry":{"type":"MultiPolygon","coordinates":77.7,12.8348],[77.6995,12.8326],[77.699,12.8305],[77.6983,12.828],[77.6978,12.8262],[77.697,12.8259],[77.6961,12.8255],[77.6951,12.825],[77.6935,12.8243],[77.6929,12.8241],[77.6893,12.8251],[77.6884,12.8253],[77.6871,12.8258],[77.6826,12.8271],[77.6829,12.8288],[77.6841,12.8313],[77.684,12.8333],[77.6838,12.8351],[77.685,12.8374],[77.6856,12.8384],[77.6863,12.84],[77.687,12.8415],[77.6885,12.8394],[77.6896,12.8392],[77.6916,12.8384],[77.6935,12.8379],[77.6954,12.8373],[77.6963,12.8369],[77.697,12.8364],[77.6979,12.8358],[77.6989,12.8351],[77.7,12.8348},"geometry_name":"the_geom","properties":{"VIL_2011":"613256","VIL_NAME11":"Kammasandra"}}],"crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:EPSG::4326"}}}
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0
>>
>> On 20 January 2017 at 16:43, Thejesh GN  wrote:
>>
>>> Seems to have only the following data for each village. Dont know
>>> what minx, maxx etc are. Is geo layer stored separately
>>>
>>>

Re: [datameet] Re: India Shape Files

[शरच्चंद्र लेले]

Dear Arun:

Can you help me understand this by downloading, processing attributes, 
and sharing the village layer for one state--say Madhya Pradesh?


I may be asking too much, but in fact if this method for downloading 
village shapefiles from Bhuvan is feasible, it would be great to 
complete the repository of village boundary shapefiles started by 
Thejesh and other datameet folks.


Sharad


On 28-10-2018 11:58, Arun Ganesh wrote:
The various village/taluk shapefile layers are available in the Bhuvan 
GeoServer layer listing:


http://bhuvannuis.nrsc.gov.in/bhuvan/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage

Search for 'vill' for village boundaries by state. Search for 'taluk' 
for all taluk boundaries. Note that the page gives a timeout error if 
you take more than a few seconds and you need to start over. Since WFS 
export is disabled, you can use the GeoRSS option to download the shape.


The one issue is that the attributes are encoded as an html string and 
needs to be parsed out. Nikhil has a great python script to do this: 
https://github.com/answerquest/asi-monuments-xml2csv/blob/master/asi-monuments2.ipynb 
but needs to be customized for each layer.




On Fri, May 11, 2018 at 1:48 PM Sharad Lele > wrote:


Hi Thejesh:

Did you ever manage to convert the data into a polygon file? I
could not follow the technical intricacies of what you were
discussing with Palash, but certainly the original json file
posted by him did not contain the polygons and could not be
imported into QGIS. Do you now have one which can be? And is it
good quality?

Sharad

On Friday, January 20, 2017 at 6:24:51 PM UTC+5:30, Thejesh GN wrote:

Imported into mongo to do some initial analysis[1].

Will do some more analysis.


> db.states.count()
36
> db.districts.count()
672
> db.taluks.count()
5960
> db.villages.count()
522291




[1]
https://gist.github.com/thejeshgn/ea0074077b6fcd6c0108b2a4380fc354

Thej
--
Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

On 20 January 2017 at 17:03, Palash Kulshrestha
 wrote:

Hi Thejesh
Thanks for the heads up, I wasn't aware of the underlying
architecture which is used in geo systems. So wasn't able
to understand the layers data. And now I am able to
understand how they come up with polygon on the map. Other
wise I was thinking that they somehow came up random
shapes using min max x,y.

I'm looking into the new api that you have provided and
see what I can come up with. I ll update. Any advice?

On Friday, January 20, 2017 at 4:54:47 PM UTC+5:30,
Thejesh GN wrote:

@Palash Kulshrestha
Let me know if you downloaded the shapes too.


The accuracy isn't bad for a national level project


https://gist.github.com/thejeshgn/2b50ce7fe45287152f55941bd8d76869





Thej
--
Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

On 20 January 2017 at 16:47, Thejesh GN
 wrote:

Ah


https://ncog.gov.in/garvgis/admin/getTalukaWiseVillageBoundary?districtId=572=05545=613256

Gives


{"gid":438129,"vilNameSoi":"KAMMASANDRA","stateId":"29","distId":"572","subdistId":"05545","villageId":"613256","villageName":"Kammasandra","minx":77.68262864449315,"miny":12.824103451365033,"maxx":77.69996896356992,"maxy":12.841455112691222}


Then you get the actual layer using


https://ncog.gov.in/garvgis/admin/ows?service=WFS=1.0.0=GetFeature=cite:Village2014=the_geom,VIL_2011,VIL_NAME11_FILTER=VIL_2011=%27613256%27=50=json

Re: [datameet] Re: India Shape Files

[Arun Ganesh]

The various village/taluk shapefile layers are available in the Bhuvan
GeoServer layer listing:

http://bhuvannuis.nrsc.gov.in/bhuvan/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage

Search for 'vill' for village boundaries by state. Search for 'taluk' for
all taluk boundaries. Note that the page gives a timeout error if you take
more than a few seconds and you need to start over. Since WFS export is
disabled, you can use the GeoRSS option to download the shape.

The one issue is that the attributes are encoded as an html string and
needs to be parsed out. Nikhil has a great python script to do this:
https://github.com/answerquest/asi-monuments-xml2csv/blob/master/asi-monuments2.ipynb
but needs to be customized for each layer.



On Fri, May 11, 2018 at 1:48 PM Sharad Lele  wrote:

> Hi Thejesh:
>
> Did you ever manage to convert the data into a polygon file? I could not
> follow the technical intricacies of what you were discussing with Palash,
> but certainly the original json file posted by him did not contain the
> polygons and could not be imported into QGIS. Do you now have one which can
> be? And is it good quality?
>
> Sharad
>
> On Friday, January 20, 2017 at 6:24:51 PM UTC+5:30, Thejesh GN wrote:
>>
>> Imported into mongo to do some initial analysis[1].
>>
>> Will do some more analysis.
>>
>>
>> > db.states.count()
>> 36
>> > db.districts.count()
>> 672
>> > db.taluks.count()
>> 5960
>> > db.villages.count()
>> 522291
>>
>>
>>
>>
>> [1]  https://gist.github.com/thejeshgn/ea0074077b6fcd6c0108b2a4380fc354
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0
>>
>> On 20 January 2017 at 17:03, Palash Kulshrestha 
>> wrote:
>>
>>> Hi Thejesh
>>> Thanks for the heads up, I wasn't aware of the underlying architecture
>>> which is used in geo systems. So wasn't able to understand the layers data.
>>> And now I am able to understand how they come up with polygon on the map.
>>> Other wise I was thinking that they somehow came up random shapes using min
>>> max x,y.
>>>
>>> I'm looking into the new api that you have provided and see what I can
>>> come up with. I ll update. Any advice?
>>>
>>> On Friday, January 20, 2017 at 4:54:47 PM UTC+5:30, Thejesh GN wrote:

 @Palash Kulshrestha
 Let me know if you downloaded the shapes too.


 The accuracy isn't bad for a national level project

 https://gist.github.com/thejeshgn/2b50ce7fe45287152f55941bd8d76869





 Thej
 --
 Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
 http://thejeshgn.com
 GPG ID :  0xBFFC8DD3C06DD6B0

 On 20 January 2017 at 16:47, Thejesh GN  wrote:

> Ah
>
>
> https://ncog.gov.in/garvgis/admin/getTalukaWiseVillageBoundary?districtId=572=05545=613256
>
> Gives
>
> {"gid":438129,"vilNameSoi":"KAMMASANDRA","stateId":"29","distId":"572","subdistId":"05545","villageId":"613256","villageName":"Kammasandra","minx":77.68262864449315,"miny":12.824103451365033,"maxx":77.69996896356992,"maxy":12.841455112691222}
>
>
> Then you get the actual layer using
>
> https://ncog.gov.in/garvgis/admin/ows?service=WFS=1.0.0=GetFeature=cite:Village2014=the_geom,VIL_2011,VIL_NAME11_FILTER=VIL_2011=%27613256%27=50=json
>
> {"type":"FeatureCollection","totalFeatures":1,"features":[{"type":"Feature","id":"Village2014.438129","geometry":{"type":"MultiPolygon","coordinates":77.7,12.8348],[77.6995,12.8326],[77.699,12.8305],[77.6983,12.828],[77.6978,12.8262],[77.697,12.8259],[77.6961,12.8255],[77.6951,12.825],[77.6935,12.8243],[77.6929,12.8241],[77.6893,12.8251],[77.6884,12.8253],[77.6871,12.8258],[77.6826,12.8271],[77.6829,12.8288],[77.6841,12.8313],[77.684,12.8333],[77.6838,12.8351],[77.685,12.8374],[77.6856,12.8384],[77.6863,12.84],[77.687,12.8415],[77.6885,12.8394],[77.6896,12.8392],[77.6916,12.8384],[77.6935,12.8379],[77.6954,12.8373],[77.6963,12.8369],[77.697,12.8364],[77.6979,12.8358],[77.6989,12.8351],[77.7,12.8348},"geometry_name":"the_geom","properties":{"VIL_2011":"613256","VIL_NAME11":"Kammasandra"}}],"crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:EPSG::4326"}}}
>
>
>
>
>
>
>
>
>
>
>
> Thej
> --
> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
> GPG ID :  0xBFFC8DD3C06DD6B0
>
> On 20 January 2017 at 16:43, Thejesh GN  wrote:
>
>> Seems to have only the following data for each village. Dont know
>> what minx, maxx etc are. Is geo layer stored separately
>>
>>
>> {"gid":14741,"vilNameSoi":"KAJLA","stateId":"19","distId":"344","subdistId":"02465","villageId":"343022","villageName":"Kajla",
>>
>> "minx":87.37861784203545,"miny":22.159016472923042,"maxx":87.38472569821403,"maxy":22.162844638588865,"type":"village"}
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0