Re: [datameet] Re: Water level data extraction

[Dilawar Singh]

Web-scraping requires knowledge of some programming language. In essence,
it is an art of transforming HTML to a format you like. For common tasks,
you can find some tools easily on the net e.g., extract all tables from
this URL and save them as csv.

For example, you can get the html from the above link using `curl` and pass
it to `pandoc` (or html2text) to convert to plain text.

[dilawars@localhost Downloads]$ curl
https://vmc.gov.in/WaterLevel.aspx | pandoc -f html -t plain
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
100 14330  100 143300 0  33325  0 --:--:-- --:--:-- --:--:-- 33325
Water Level Statistics

Ajwa

Vishwamitri

  Date  TimeLevel (Feet)
   --- --
   17/06/2020   06:00  207.65
   17/06/2020   03:00  207.55
   17/06/2020   00:00  207.55

  Date  TimeLevel (Feet)
   --- --
   17/06/2020   06:00   4.00
   16/06/2020   21:00   4.00
   16/06/2020   18:00   4.00


Copy paste this in a Libreoffice-Calc or MS OFFICE spreadsheet and see if
it gets converted to a spreadsheet.

Anyway, this webpage does accept a POST request but only in encrypted
format. So it is essentially useless. Contact the author of
https://craigdsouza.github.io/data/IMD-Hourly-Precipitation-Data, he
scraped rainfall data from IMD website (2018). This data is probably of no
use to you but he might know a better source to get this data. I couldn't
locate time series data for a city on IMD website.

best,
   Dilawar



On Tue, Jun 16, 2020 at 10:42 PM Vikas Rana  wrote:

> I don't know how to scrape the data... Any guidance would be helpful.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/a5f65448-059b-4132-828c-ff36ccc60310o%40googlegroups.com
> .
>


-- 
Dilawar
NCBS Bangalore

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAM72-Zv6EybsrbjvLq9-%2BpuQK3L-Ypf0dD92JZGEBGBD0Xdr9Q%40mail.gmail.com.

[datameet] Re: Officially Approved India Map Borders?

[Dilawar Singh]

http://www.surveyofindia.gov.in/pages/display/122-outline-maps-of-india

Example PDF (it is a scan copy): 
http://www.surveyofindia.gov.in/files/36m_india.pdf . Most maps on this 
site are not machine readable. 

best,
   Dilawar 



On Tuesday, June 16, 2020 at 10:49:46 PM UTC+5:30, Curran Kelleher wrote:
>
> Greetings,
>
> I'm working on some mapping projects where the goal is to be able to 
> publish the map legally within India. To do this, we need to use the "Gov't 
> of India - Approved" borders for India (particularly disputed borders). The 
> closest we've come is to use GADM India Border 
> , but this is not officially approved by 
> Gov't of India, and makes some comprimises around disputed borders. We have 
> also used Natural Earth Disputed Borders 
> 
>  
> as an overlay, but this is also not the ideal solution.
>
> So, does anyone know how to find an officially approved map of India's 
> borders? Ideally machine readable (Shapefile, GeoJSON), but alternatively 
> as an image? A PDF? Even can you get any printed physical maps that are 
> officially approved? I have not been able to find these maps anywhere. Many 
> thanks in advance.  
> 
>
> Best regards,
> Curran
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/b8f797b8-0ab0-4bb4-9466-0389b51b4b0eo%40googlegroups.com.

Re: [datameet] Officially Approved India Map Borders?

[Thejesh GN]

https://github.com/datameet/maps/tree/master/Country


https://thejeshgn.com/2018/11/15/download-official-external-boundaries-of-india-by-survey-of-india/

Regards,
Thej

Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್
http://thejeshgn.com

On Tue, 16 Jun, 2020, 10:49 PM Curran Kelleher, 
wrote:

> Greetings,
>
> I'm working on some mapping projects where the goal is to be able to
> publish the map legally within India. To do this, we need to use the "Gov't
> of India - Approved" borders for India (particularly disputed borders). The
> closest we've come is to use GADM India Border
> , but this is not officially approved by
> Gov't of India, and makes some comprimises around disputed borders. We have
> also used Natural Earth Disputed Borders
> 
> as an overlay, but this is also not the ideal solution.
>
> So, does anyone know how to find an officially approved map of India's
> borders? Ideally machine readable (Shapefile, GeoJSON), but alternatively
> as an image? A PDF? Even can you get any printed physical maps that are
> officially approved? I have not been able to find these maps anywhere. Many
> thanks in advance. 
> 
>
> Best regards,
> Curran
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/395b8264-b1fb-4a2e-b731-02cd74364395o%40googlegroups.com
> 
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAABnYsXU0-45zZjQ7MZ8%2BGw9MuAH7-Dqr8ntoTMsrAN6zmf3hA%40mail.gmail.com.

Re: [datameet] Officially Approved India Map Borders?

[Primal Vincent]

Hi Curran,

This is 2019 updated map.

https://www.amcharts.com/svg-maps/?map=india2019

Included with Latest UT Ladakh & Jammu Kashmir

Hope this helps
Primal

On Tue, Jun 16, 2020 at 1:19 PM, Curran Kelleher 
wrote:

> Greetings,
>
> I'm working on some mapping projects where the goal is to be able to
> publish the map legally within India. To do this, we need to use the "Gov't
> of India - Approved" borders for India (particularly disputed borders). The
> closest we've come is to use GADM India Border
> , but this is not officially approved by
> Gov't of India, and makes some comprimises around disputed borders. We have
> also used Natural Earth Disputed Borders
> 
> as an overlay, but this is also not the ideal solution.
>
> So, does anyone know how to find an officially approved map of India's
> borders? Ideally machine readable (Shapefile, GeoJSON), but alternatively
> as an image? A PDF? Even can you get any printed physical maps that are
> officially approved? I have not been able to find these maps anywhere. Many
> thanks in advance. 
> 
>
> Best regards,
> Curran
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/395b8264-b1fb-4a2e-b731-02cd74364395o%40googlegroups.com
> 
> .
>
-- 
Primal K Vincent
Mob # : 90481 49964

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAPRin7FXhM6ujTrd56TV9QSGcGO-XRr_s2pBdD-0ykY0e%3DEw%3Dg%40mail.gmail.com.

[datameet] Officially Approved India Map Borders?

[Curran Kelleher]

Greetings,

I'm working on some mapping projects where the goal is to be able to 
publish the map legally within India. To do this, we need to use the "Gov't 
of India - Approved" borders for India (particularly disputed borders). The 
closest we've come is to use GADM India Border 
, but this is not officially approved by 
Gov't of India, and makes some comprimises around disputed borders. We have 
also used Natural Earth Disputed Borders 

 
as an overlay, but this is also not the ideal solution.

So, does anyone know how to find an officially approved map of India's 
borders? Ideally machine readable (Shapefile, GeoJSON), but alternatively 
as an image? A PDF? Even can you get any printed physical maps that are 
officially approved? I have not been able to find these maps anywhere. Many 
thanks in advance.  


Best regards,
Curran

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/395b8264-b1fb-4a2e-b731-02cd74364395o%40googlegroups.com.

[datameet] Re: Water level data extraction

[Vikas Rana]

I don't know how to scrape the data... Any guidance would be helpful.

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/a5f65448-059b-4132-828c-ff36ccc60310o%40googlegroups.com.

[datameet] Re: Water level data extraction

[Palash Kulshrestha]

in wayback machine there is data for 12 august
https://web.archive.org/web/20190812194137/https://vmc.gov.in/WaterLevel.aspx

On Tuesday, June 16, 2020 at 10:10:58 PM UTC+5:30, Palash Kulshrestha wrote:
>
> i dont think there is any option to retrieve old data. i can help you to 
> scrape if you know how .
>
> On Thursday, June 11, 2020 at 5:25:38 PM UTC+5:30, Vikas Rana wrote:
>>
>> Is there any way by which we can extract the data for all the dates from 
>> https://vmc.gov.in/waterlevel.aspx website? Actually, I want  28 July 
>> 2019 to 2 august 2019 data for a 2-dimensional flood simulation, but the 
>> extraction of all the data would be much more beneficial. Thank you.
>>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/038419ac-866c-45cd-8a14-609b565ddf15o%40googlegroups.com.

[datameet] Re: Water level data extraction

[Palash Kulshrestha]

i dont think there is any option to retrieve old data. i can help you to 
scrape if you know how .

On Thursday, June 11, 2020 at 5:25:38 PM UTC+5:30, Vikas Rana wrote:
>
> Is there any way by which we can extract the data for all the dates from 
> https://vmc.gov.in/waterlevel.aspx website? Actually, I want  28 July 
> 2019 to 2 august 2019 data for a 2-dimensional flood simulation, but the 
> extraction of all the data would be much more beneficial. Thank you.
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/9a7070f0-7168-4a91-87e6-b724adb095afo%40googlegroups.com.

Re: [datameet] Re: Open Data in India: In a Restrictive Copyright Regime, Voluntary Organisations Pitch in to Make Data Accessible

[Dilawar Singh]

Hi Thej, Datameet

After reading this, I went to SOI site and tried downloading few maps. I
could only find PDFs. And data.gov.in also love PDFs.

I wrote a couple of scripts which I used to figure out boundaries  and
extract data from curves in old PDF paper. It's a tricky business but I had
decent success (One python3 app is here
https://github.com/dilawar/PlotDigitizer). These scripts can also be
tweaked to fetch boundaries from PDF maps (no deep-learning business yet).
If datameet still struggling to find the boundaries from such maps, please
send a few samples to me. I can probably setup a web-service to upload the
map and get the GeoJSON if results are OK.

best,
Dilawar

On Tue, Jun 16, 2020 at 8:49 AM Naveen Francis  wrote:

> Nice writeup Thej,
>
> SOI point of view is completely different
> https://youtu.be/o1qMZZOia3c?t=3174
>
> 'Don't take data, but starting using SOI services.'
>
>
> On Monday, 15 June 2020 11:11:43 UTC+5:30, Thejesh GN wrote:
>>
>> Team,
>>
>> I wrote a piece for EPW about the status of Open Data and How communities
>> like ours. Thank you for being an awesome community. The article is open to
>> all.
>>
>> Open Data in India: In a Restrictive Copyright Regime, Voluntary
>> Organisations Pitch in to Make Data Accessible
>>
>> 
>>
>> https://www.epw.in/engage/article/voluntary-organisations-india-counteract-states-copyright-regime-open-data
>> 
>>
>> - This was pre covid-19, so a lot of things related to that are missing.
>> - Comments are welcome here on this thread or  on this blog post
>> 
>>
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/a5569aa9-6842-4a90-9127-33c9f603af50o%40googlegroups.com
> 
> .
>


-- 
Dilawar
NCBS Bangalore

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAM72-ZuRcATxrShC8zBtGVofQ4Ku37o0E5qbVBAZUdZtXLV2%2Bg%40mail.gmail.com.

Re: [datameet] Re: Open Data in India: In a Restrictive Copyright Regime, Voluntary Organisations Pitch in to Make Data Accessible

[Natasha kalra]

Hi all

Is there any population-wise data available at one place for all the ULBs
which participated in Swachh Bharat ranking 2019?

Thanks in advance
Natasha



On Tue, Jun 16, 2020 at 8:49 AM Naveen Francis  wrote:

> Nice writeup Thej,
>
> SOI point of view is completely different
> https://youtu.be/o1qMZZOia3c?t=3174
>
> 'Don't take data, but starting using SOI services.'
>
>
> On Monday, 15 June 2020 11:11:43 UTC+5:30, Thejesh GN wrote:
>>
>> Team,
>>
>> I wrote a piece for EPW about the status of Open Data and How communities
>> like ours. Thank you for being an awesome community. The article is open to
>> all.
>>
>> Open Data in India: In a Restrictive Copyright Regime, Voluntary
>> Organisations Pitch in to Make Data Accessible
>>
>> 
>>
>> https://www.epw.in/engage/article/voluntary-organisations-india-counteract-states-copyright-regime-open-data
>> 
>>
>> - This was pre covid-19, so a lot of things related to that are missing.
>> - Comments are welcome here on this thread or  on this blog post
>> 
>>
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/a5569aa9-6842-4a90-9127-33c9f603af50o%40googlegroups.com
> 
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAM2y7dq9LD1jLH_n%3Dq4z%2Bi1mv37z8h%3D6RnQF7skumZF2x830zw%40mail.gmail.com.