Re: [datameet] Help with schools location data extracting

[Devdatta Tengshe]

Hi,

Since this is an ArcGIS Server REST Service, one needs to be familiar with
the REST API: (
https://developers.arcgis.com/rest/services-reference/map-service.htm)

Then One needs to know which layer/MapService is serving out the data that
you need.
And finally, you need to query the 'Query
'
end point of the appropriate end point


Regards,
Devdatta


On Sun, Dec 23, 2018 at 8:09 PM Naraina Damle  wrote:

> Hello,
>
> What do I need to do if I want to extract the Name of School and Lat Lon
> in a particular area?
>
> On Sunday, May 22, 2016 at 6:14:31 PM UTC+5:30, Devdatta Tengshe wrote:
>>
>> Nihkil,
>>
>> There is a standard ArcGIS rest endpoint which is the backend for this
>> here: http://rsgis.nic.in/ArcGIS/rest/services/?f=json.
>>
>> What data do you want to extract?
>>
>> Regards,
>> Devdatta
>>
>> On Sat, May 21, 2016 at 4:04 PM, Venkata Pingali 
>> wrote:
>>
>>> I looked at this site some months back. Notes from my investigation
>>> then:
>>>
>>> 1. The content/jsons are being served by commercial GIS operated by BSNL
>>> 2. The content is coming back split across many JSONs (50-60) with
>>> encoded URLs
>>> 3. API appeared stateful (URLs kept changing)
>>>
>>> This, I concluded, needs a
>>> (a) phantom-like programmable browser control
>>> (b) network proxy/sniffer to dump all intercepted content
>>> (c) wrangling to relate data from across all the jsons
>>>
>>> I also believe that there are license issues.
>>>
>>> On May 21, 2016 3:51 PM, "Raphael Susewind" 
>>> wrote:
>>>
 Hi Nikhil,

 most likely the flash application loads something like a JSON (or CSV,
 if they are bad programmers ;-) ) from a specified API address. Use a
 network sniffer to intercept the traffic that the flashplayer generates,
 and see whether you can replicate the API.

 If you are lucky, you will see HTTP requests to an URL along the lines
 of http://schoolgis.nic.in/state_x/data.json?school=001 to
 14986. In that case, you can then manually scrape the JSON files (if
 need be by emulating a flashplayer's HTTP headers, though I doubt that
 they check for this).

 If you are unlucky, its a more complex API - some stateful frontends for
 SQL databases can be very nasty to replicate, for instance. One brute
 force kind of solution in such cases would be to write a custom proxy
 server (there are python/perl/... modules for this) - i.e. a kind of
 customized sniffer - and route your browser traffic through this, then
 automate the browser (again, there are plugins for firefox and chrome
 that have corresponding python or perl interfaces), and intercept the
 traffic generated. That's the solution I found to scrape polling station
 localities from the ECI server (before they put a bold copyright
 disclaimer on it - now this kind of scraping would probably be illegal -
 so do check these issues as well).

 Let us know what you find out about the API,

 Best of luck,
 Raphael

 On 21.05.2016 11:43, Nikhil VJ wrote:

 > Hi friends,
 >
 > is their any way to extract data from such a flash player
 platform...as
 > follows...
 >
 > schoolgis.nic.in 
 >
 > --regards,
 > Nikhil VJ
 > Pune
 >
 > --
 > Datameet is a community of Data Science enthusiasts in India. Know
 more
 > about us by visiting http://datameet.org
 > ---
 > You received this message because you are subscribed to the Google
 > Groups "datameet" group.
 > To unsubscribe from this group and stop receiving emails from it, send
 > an email to datameet+u...@googlegroups.com
 > .
 > For more options, visit https://groups.google.com/d/optout.

 --
 Dr Raphael Susewind | Associate, Contemporary South Asia Studies, Oxford
  Snail Mail | Melanchthonstr. 4a, 33615 Bielefeld, Germany
   Web & Twitter | https://www.raphael-susewind.de |
 @RaphaelSusewind
  Impact | https://impactstory.org/raphael-susewind

 Please consider https://www.gnupg.org for encryption (key id 10AEE42F)

 --
 Datameet is a community of Data Science enthusiasts in India. Know more
 about us by visiting http://datameet.org
 ---
 You received this message because you are subscribed to the Google
 Groups "datameet" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to datameet+u...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.

>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To 

Re: [datameet] Help with schools location data extracting

[Naraina Damle]

Hello, 

What do I need to do if I want to extract the Name of School and Lat Lon in 
a particular area?

On Sunday, May 22, 2016 at 6:14:31 PM UTC+5:30, Devdatta Tengshe wrote:
>
> Nihkil,
>
> There is a standard ArcGIS rest endpoint which is the backend for this 
> here: http://rsgis.nic.in/ArcGIS/rest/services/?f=json.
>
> What data do you want to extract?
>
> Regards,
> Devdatta
>
> On Sat, May 21, 2016 at 4:04 PM, Venkata Pingali  > wrote:
>
>> I looked at this site some months back. Notes from my investigation then: 
>>
>> 1. The content/jsons are being served by commercial GIS operated by BSNL
>> 2. The content is coming back split across many JSONs (50-60) with 
>> encoded URLs 
>> 3. API appeared stateful (URLs kept changing) 
>>
>> This, I concluded, needs a 
>> (a) phantom-like programmable browser control 
>> (b) network proxy/sniffer to dump all intercepted content 
>> (c) wrangling to relate data from across all the jsons 
>>
>> I also believe that there are license issues. 
>>
>> On May 21, 2016 3:51 PM, "Raphael Susewind" > > wrote:
>>
>>> Hi Nikhil,
>>>
>>> most likely the flash application loads something like a JSON (or CSV,
>>> if they are bad programmers ;-) ) from a specified API address. Use a
>>> network sniffer to intercept the traffic that the flashplayer generates,
>>> and see whether you can replicate the API.
>>>
>>> If you are lucky, you will see HTTP requests to an URL along the lines
>>> of http://schoolgis.nic.in/state_x/data.json?school=001 to
>>> 14986. In that case, you can then manually scrape the JSON files (if
>>> need be by emulating a flashplayer's HTTP headers, though I doubt that
>>> they check for this).
>>>
>>> If you are unlucky, its a more complex API - some stateful frontends for
>>> SQL databases can be very nasty to replicate, for instance. One brute
>>> force kind of solution in such cases would be to write a custom proxy
>>> server (there are python/perl/... modules for this) - i.e. a kind of
>>> customized sniffer - and route your browser traffic through this, then
>>> automate the browser (again, there are plugins for firefox and chrome
>>> that have corresponding python or perl interfaces), and intercept the
>>> traffic generated. That's the solution I found to scrape polling station
>>> localities from the ECI server (before they put a bold copyright
>>> disclaimer on it - now this kind of scraping would probably be illegal -
>>> so do check these issues as well).
>>>
>>> Let us know what you find out about the API,
>>>
>>> Best of luck,
>>> Raphael
>>>
>>> On 21.05.2016 11:43, Nikhil VJ wrote:
>>>
>>> > Hi friends,
>>> >
>>> > is their any way to extract data from such a flash player platform...as
>>> > follows...
>>> >
>>> > schoolgis.nic.in 
>>> >
>>> > --regards,
>>> > Nikhil VJ
>>> > Pune
>>> >
>>> > --
>>> > Datameet is a community of Data Science enthusiasts in India. Know more
>>> > about us by visiting http://datameet.org
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> > Groups "datameet" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an email to datameet+u...@googlegroups.com 
>>> > .
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> Dr Raphael Susewind | Associate, Contemporary South Asia Studies, Oxford
>>>  Snail Mail | Melanchthonstr. 4a, 33615 Bielefeld, Germany
>>>   Web & Twitter | https://www.raphael-susewind.de | @RaphaelSusewind
>>>  Impact | https://impactstory.org/raphael-susewind
>>>
>>> Please consider https://www.gnupg.org for encryption (key id 10AEE42F)
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more 
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google 
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to datameet+u...@googlegroups.com .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> -- 
>> Datameet is a community of Data Science enthusiasts in India. Know more 
>> about us by visiting http://datameet.org
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to datameet+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [datameet] Re: Pune Tree Census site

[Devdatta Tengshe]

The WMS service is currently not reponding, because I'm currently
'load-testing' it while scraping the data. You can try it in some time.


Regards,
Devdatta


On Sun, Dec 23, 2018 at 3:19 PM Arun Ganesh  wrote:

>
>>
>> You can add this as a Layer to Qgis, using the following URL:
>> http://103.233.79.142:90/geoserver/ows?service=wms=1.1.1=GetCapabilities
>>
>> I get connection refused when I try to add the WMS layer in QGIS. Is
> there any authentication needed?
>
> Also, theres a plan for a citizen led tree survey in in Bengaluru. It
> would be good to know how the Pune tree survey was done and if the same
> methodology and tools could be used.
> https://www.deccanherald.com/city/indiranagar-residents-map-705806.html
>
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [datameet] Re: Pune Tree Census site

[Nikhil VJ]

Hi Devdatta,

Wow, that's great!

I added the url in QGIS, pressed Connect, it showed several layers, I chose 
"tlpmc trees_display" as that's mentioned in the XHR data,
And it's coming as "Connection refused" each time.

Are there more settings to be set here? See screenshot 
. (never worked with WMS stuff yet)

-Nikhil

On Sunday, December 23, 2018 at 2:35:58 PM UTC+5:30, Devdatta Tengshe wrote:
>
> Hi Nikhil,
>
> The backend is a Geoserver, which uses WMS & WFS services.
>
> You can add this as a Layer to Qgis, using the following URL: 
> http://103.233.79.142:90/geoserver/ows?service=wms=1.1.1=GetCapabilities
>
> They have also kept the WFS service running, so it should be possible to 
> extract the data.
>
> Here is sample in CSV format.
>
>
> Regards,
> Devdatta
>
>
> On Sun, Dec 23, 2018 at 12:50 PM Nikhil VJ  > wrote:
>
>> Pasting details of an XHR request triggered by zooming into the map and 
>> clicking on location of one tree. This was captured from browser console > 
>> Network tab. If we click at a zoomed-out level on a bunch then it gives 
>> details of max 50 trees found around that point, so I'm suspecting its a 
>> radius search. Please check and tell if the URL doesn't return anything at 
>> your end. 
>>
>> link:
>>
>> http://103.233.79.142:90/geoserver/wms?SERVICE=WMS=1.1.1=GetFeatureInfo=image%2Fpng=true_LAYERS=tlpmc%3Atrees_display=false=tlpmc%3Atrees=tlpmc%3Atrees_display_FILTER=1%3D1_FORMAT=application%2Fjson_COUNT=50=180=236=256=256=EPSG%3A3857=8213540.8743836135%2C2100986.3779604957%2C8213579.092897756%2C2101024.596474638
>>
>> GET parameters laid out:
>> SERVICE=WMS
>> VERSION=1.1.1
>> REQUEST=GetFeatureInfo
>> FORMAT=image%2Fpng
>> TRANSPARENT=true
>> QUERY_LAYERS=tlpmc%3Atrees_display
>> tiled=false
>> STYLES=tlpmc%3Atrees
>> LAYERS=tlpmc%3Atrees_display
>> CQL_FILTER=1%3D1
>> INFO_FORMAT=application%2Fjson
>> FEATURE_COUNT=50
>> X=180
>> Y=236
>> WIDTH=256
>> HEIGHT=256
>> SRS=EPSG%3A3857
>>
>> BBOX=8213540.8743836135%2C2100986.3779604957%2C8213579.092897756%2C2101024.596474638
>>
>>
>> Response:
>> {
>>   "type": "FeatureCollection",
>>   "totalFeatures": "unknown",
>>   "features": [
>> {
>>   "type": "Feature",
>>   "id": "trees_display.fid-66dc9168_167d97e634c_-5ffe",
>>   "geometry": {
>> "type": "Point",
>> "coordinates": [
>>   8213567.560115479,
>>   2100989.363806502
>> ]
>>   },
>>   "geometry_name": "geom",
>>   "properties": {
>> "id": 3720853,
>> "oid": null,
>> "sr_no": null,
>> "girth_cm": 30,
>> "height_m": 7,
>> "canopy_dia_m": 3,
>> "condition": "Healthy",
>> "other_remarks": null,
>> "ownership": "Private",
>> "society_name": "NA",
>> "road_name": "Pashan Sutarwadi Link Road",
>> "northing": 18.5411655788,
>> "easting": 73.7837327644,
>> "balanced": true,
>> "remarks": "NA",
>> "special_collar": "N/A",
>> "ward_name": "10",
>> "botanical_name": "Mangifera indica Linn.",
>> "saar_uid": "2605965",
>> "common_name": "Mango",
>> "local_name": "Amba",
>> "economic_i": "Fruit",
>> "phenology": "Seasonal",
>> "flowering": "January-March",
>> "ward": 10,
>> "is_rare": false
>>   }
>> }
>>   ],
>>   "crs": {
>> "type": "name",
>> "properties": {
>>   "name": "urn:ogc:def:crs:EPSG::3857"
>> }
>>   }
>> }
>>
>> Also, query for GIS folks (believe it or not, this mailing list isn't 
>> exclusively about GIS only!): is the WMS url in a familiar format that can 
>> be brought into QGIS or the like?
>>
>> -- 
>> Datameet is a community of Data Science enthusiasts in India. Know more 
>> about us by visiting http://datameet.org
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to datameet+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [datameet] Re: Pune Tree Census site

[Arun Ganesh]

>
>
>
> You can add this as a Layer to Qgis, using the following URL:
> http://103.233.79.142:90/geoserver/ows?service=wms=1.1.1=GetCapabilities
>
> I get connection refused when I try to add the WMS layer in QGIS. Is there
any authentication needed?

Also, theres a plan for a citizen led tree survey in in Bengaluru. It would
be good to know how the Pune tree survey was done and if the same
methodology and tools could be used.
https://www.deccanherald.com/city/indiranagar-residents-map-705806.html

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [datameet] Re: Pune Tree Census site

[Devdatta Tengshe]

Hi Nikhil,

The backend is a Geoserver, which uses WMS & WFS services.

You can add this as a Layer to Qgis, using the following URL:
http://103.233.79.142:90/geoserver/ows?service=wms=1.1.1=GetCapabilities

They have also kept the WFS service running, so it should be possible to
extract the data.

Here is sample in CSV format.


Regards,
Devdatta


On Sun, Dec 23, 2018 at 12:50 PM Nikhil VJ  wrote:

> Pasting details of an XHR request triggered by zooming into the map and
> clicking on location of one tree. This was captured from browser console >
> Network tab. If we click at a zoomed-out level on a bunch then it gives
> details of max 50 trees found around that point, so I'm suspecting its a
> radius search. Please check and tell if the URL doesn't return anything at
> your end.
>
> link:
>
> http://103.233.79.142:90/geoserver/wms?SERVICE=WMS=1.1.1=GetFeatureInfo=image%2Fpng=true_LAYERS=tlpmc%3Atrees_display=false=tlpmc%3Atrees=tlpmc%3Atrees_display_FILTER=1%3D1_FORMAT=application%2Fjson_COUNT=50=180=236=256=256=EPSG%3A3857=8213540.8743836135%2C2100986.3779604957%2C8213579.092897756%2C2101024.596474638
>
> GET parameters laid out:
> SERVICE=WMS
> VERSION=1.1.1
> REQUEST=GetFeatureInfo
> FORMAT=image%2Fpng
> TRANSPARENT=true
> QUERY_LAYERS=tlpmc%3Atrees_display
> tiled=false
> STYLES=tlpmc%3Atrees
> LAYERS=tlpmc%3Atrees_display
> CQL_FILTER=1%3D1
> INFO_FORMAT=application%2Fjson
> FEATURE_COUNT=50
> X=180
> Y=236
> WIDTH=256
> HEIGHT=256
> SRS=EPSG%3A3857
>
> BBOX=8213540.8743836135%2C2100986.3779604957%2C8213579.092897756%2C2101024.596474638
>
>
> Response:
> {
>   "type": "FeatureCollection",
>   "totalFeatures": "unknown",
>   "features": [
> {
>   "type": "Feature",
>   "id": "trees_display.fid-66dc9168_167d97e634c_-5ffe",
>   "geometry": {
> "type": "Point",
> "coordinates": [
>   8213567.560115479,
>   2100989.363806502
> ]
>   },
>   "geometry_name": "geom",
>   "properties": {
> "id": 3720853,
> "oid": null,
> "sr_no": null,
> "girth_cm": 30,
> "height_m": 7,
> "canopy_dia_m": 3,
> "condition": "Healthy",
> "other_remarks": null,
> "ownership": "Private",
> "society_name": "NA",
> "road_name": "Pashan Sutarwadi Link Road",
> "northing": 18.5411655788,
> "easting": 73.7837327644,
> "balanced": true,
> "remarks": "NA",
> "special_collar": "N/A",
> "ward_name": "10",
> "botanical_name": "Mangifera indica Linn.",
> "saar_uid": "2605965",
> "common_name": "Mango",
> "local_name": "Amba",
> "economic_i": "Fruit",
> "phenology": "Seasonal",
> "flowering": "January-March",
> "ward": 10,
> "is_rare": false
>   }
> }
>   ],
>   "crs": {
> "type": "name",
> "properties": {
>   "name": "urn:ogc:def:crs:EPSG::3857"
> }
>   }
> }
>
> Also, query for GIS folks (believe it or not, this mailing list isn't
> exclusively about GIS only!): is the WMS url in a familiar format that can
> be brought into QGIS or the like?
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


trees_display.csv
Description: MS-Excel spreadsheet

[datameet] Re: Ideas - If someone was to visit every rural road constructed in India what info would you collect?

[Nikhil VJ]

Hi Harsh,

This would be an amazing exercise indeed, and thanks so much for posting 
this open-ended question here. Sorry I didn't see this last month.
Here's my wishlist:

1. Bus stop locations
2. Crossroad / T-point / Fork locations. These can help in figuring out the 
network in form of nodes.
3. 360-degree photos. Details on that below.

*360-degree photos:*
An NGO in Pune recently got a 360-degree camera . We 
spent a day playing with it, and learned how to post its photos and videos 
online for 360-degree viewing. It's tiny and fast, and has a cool 
remote-control feature from phone (so you can hoist it on a pole, see 
preview on tablet/phone and click a pic!).

360-degree shots can go a long way in resolving mapping confusions. 
Mapillary  site is hosting such photos and has made 
them freely available for bringing on as a layer in OpenStreetMap.org when 
we are doing mapping. Google Earth also features a layer from 
360cities.net, one can sign up there and upload shots.

The 360-degree shot will capture a LOT of local visual information about 
the place that you can sit on later and churn - manually or feed it into a 
program.

One drawback we found with our device (Theta S) was that we couldn't 
capture lat-long information with the photo itself. Rather, that would have 
to captured on a phone or something, and then the photo and location would 
need to be matched later. Also, this camera's photo quality wasn't good at 
high zooms - it may have been in the lower range, and there may be 
higher-quality cameras around.



*GPS tracking:*
Note that phone GPS's are notoriously inaccurate - it'll be a good 
investment to purchase dedicated GPS devices for this survey that can 
"plug" into a phone / tablet via cord or bluetooth and become the GPS 
source. Linking some articles than may help in this regard.

https://mashtips.com/best-external-gps-receivers/amp/

http://nerdynerdnerdz.com/4053/pair-external-bluetooth-gps-receiver-with-android-for-superior-navigation/
https://bigdanzblog.wordpress.com/2015/01/18/connecting-u-blox-neo-6m-gps-to-raspberry-pi/
 
-> if someone wants to go full IoT.


All the best for this project, and try to get something released as open 
data!

Nikhil VJ
Pune, India


On Wednesday, November 28, 2018 at 5:29:24 PM UTC+5:30, Harsh Nisar wrote:
>
> Hello,
>
> There is a proposed exercise in which every rural road constructed under 
> PMGSY in India is going to be visited for *something*.
>
> As you can imagine it's a massive exercise - what additional data can be 
> collected during the visit which could be useful for any future evaluation, 
> machine learning problem , or something else creative etc.
>
> Eg. visual inspection of road,  streetlights, road side plantation etc. 
> It's only an auxilliary data collection so it can't be intensive, time 
> consuming or require specific training.
>
> Additionally, each roads satellite imagery is being looked at by a manual 
> person. What can be done in this case as well? Data annotation etc? Putting 
> it differently, what else would you ask this person to do while looking at 
> each road's imagery?
>
> Happy to provide more clarity off-thread.
>
> Regards,
> Harsh
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.