Dear Nikhil I am trying to download temperature data for 2000 of Assam
and getting this massage: {"success":false,"message":"No data for give
location / year"}. What is problem Please how to download temperature data
at 0.5 degree for all grid point of assam
On Fri, Oct 14, 2022 at 5:41 PM Nikhil VJ <nikhil...@gmail.com> wrote:
> Hi All,
>
> Got a request regd need to simplify this IMD grid data fetching process
> for folks, and that gave me impetus to complete setting up a website where
> individual location + year data can be downloaded as simple flat CSV (which
> opens in Excel / LibreOffice Calc).
>
> So here it is: *https://server.nikhilvj.co.in/imd_data/
> <https://server.nikhilvj.co.in/imd_data/>*
>
> There's a map and year selector to help choose, after that a link is
> generated for downloading. Open that in browser or do wget/curl
> on commandline, and it'll download a CSV with proper naming.
>
> Sample link:
>
> https://server.nikhilvj.co.in/imd_data/API/fetchData?year=2010&lat=25.25&lon=76.25&which=all
>
> You are *encouraged to scrape* : use the format above to compose your own
> link. (Hint: Excel formulas like: =A2&B2&"?year="&C2 are perfect for this)
>
> With a caveat: Given that it's not some fancy server that can take huge
> loads, there's a cruel and harsh rate limiter in place which will allow
> only 1 hit per 10 secs per IP address for anonymous users.
>
> If you want to bypass that, there's an email-otp login process given which
> will add a token and let you do 5 hits per second, and that won't be
> restricted by IP address.
> This would be useful in institutions where multiple users on the same
> network may want to fetch data at the same time.
> ex: a school/college hands-on session in climate data visualization where
> the host as usual forgets to pre-download the data
> So you can scrape and I can keep my server from crashing - both sides win.
>
> Source code is at https://github.com/answerquest/IMD-grid-data-work (see
> imd_data_api folder)
>
>
> --
> Cheers,
> Nikhil VJ
> https://nikhilvj.co.in
>
>
> On Wed, Oct 12, 2022 at 8:30 AM Nikhil VJ <nikhil...@gmail.com> wrote:
>
>> Hi Pankaj,
>>
>> The place initially referred to is archival data - by definition it's
>> historic and not realtime.
>> What I do see on https://imdpune.gov.in/lrfindex.php is "Gridded Data
>> Real Time" -> you might want to explore that.
>> Over there, Daily 0.5x0.5 grid temp data links are there.
>> You can select a single date there and download the file.
>>
>> I don't know right now how to open it, though. imdlib library that I've
>> used - I didn't see anything about daily data.
>> Here's its documentation:
>> https://imdlib.readthedocs.io/en/latest/Usage.html
>>
>> I've posted a query about it in the lib's github :
>> https://github.com/iamsaswata/imdlib/issues/17 , let's see if the author
>> responds. Perhaps some inspection of the source code also might yield a way
>> to open the daily data.
>>
>> PS: Next time do initiate a separate conversation with a different
>> subject line - as here you have asked about temperature data but the
>> subject reads rainfall data. Becomes less discoverable over the open
>> internet or after a long period of time.
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>> On Thu, Oct 6, 2022 at 11:56 PM Pankaj pandey <pkpner...@gmail.com>
>> wrote:
>>
>>> Thanks Akshay for sharing the code. Actually I want to download tmax and
>>> tmin data at 0.5 degree not at 1 degree for 2000 to 2021. As I want to
>>> dowanload gridded data real time Download Daily Maximum Temperature
>>> data ( 0.5 x 0.5 degree) Binary File (Real-time). In this page only the
>>> select data option is available. Should i replace year with data in code.
>>> It is requested please modify code for this case
>>>
>>> with regards
>>> p.k.pANDEY
>>>
>>>
>>>
>>> On Thu, Oct 6, 2022 at 11:30 PM Akshay Kumar <akshay.cg...@gmail.com>
>>> wrote:
>>>
>>>> Dear Sir
>>>> Pl check this link.
>>>>
>>>> https://github.com/Manisht9/imdData
>>>>
>>>>
>>>> On Thu, 6 Oct 2022 at 10:55 PM, Pankaj pandey <pkpner...@gmail.com>
>>>> wrote:
>>>>
>>>>> Dear Nikhil how can batch download data,
>>>>> https://imdpune.gov.in/lrfindex.php. This page give data wise data.
>>>>> how can download suppose 20 year data of temperature. Any python code to
>>>>> do this to bulk download in csv. Please update any code will very much
>>>>> neede
>>>>>
>>>>> With Regards
>>>>> P.K.Pandey
>>>>>
>>>>>
>>>>> On Thu, Oct 6, 2022 at 7:28 PM Nikhil VJ <nikhil...@gmail.com> wrote:
>>>>>
>>>>>> Hi Georgelio, Vastav,
>>>>>>
>>>>>> While this link is down:
>>>>>> https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html
>>>>>>
>>>>>> Upon browsing around the IMD site I found under Climate Monitoring in
>>>>>> top menu, https://imdpune.gov.in/lrfindex.php
>>>>>> Side menu: Gridded Data Archive -> All the data is there and
>>>>>> available just fine.
>>>>>>
>>>>>>
>>>>>> So, taking offline my data dump to save my server's bandwidth.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Nikhil VJ
>>>>>> https://nikhilvj.co.in
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 6, 2022 at 6:38 PM Nikhil VJ <nikhil...@gmail.com> wrote:
>>>>>>
>>>>>>> Wow, Dead Internet Theory vibes 🤐
>>>>>>>
>>>>>>> Ok, I've moved all the gridded data file copies that I had at my end
>>>>>>> to: https://server.nikhilvj.co.in/dump/imd_grid/
>>>>>>>
>>>>>>> Disclaimer: they were download from <currently down> source
>>>>>>> <https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html>
>>>>>>> on 13th July 2022, and their sha256 checksums are at:
>>>>>>>
>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_rain.txt
>>>>>>>
>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmax.txt
>>>>>>>
>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmin.txt
>>>>>>>
>>>>>>> Beyond that digital trail, no guarantee on integrity of the data
>>>>>>> from my side. Use at own risk.
>>>>>>>
>>>>>>> This code snippet should help you turn a file into a flat CSV table:
>>>>>>>
>>>>>>> import imdlib
>>>>>>> rain1 = imdlib.open_data('rain', 2010, 2010,
>>>>>>> 'yearwise').get_xarray().to_dataframe()
>>>>>>> rain2 = rain1[rain1['rain'] > -100].reset_index()
>>>>>>> rain2.to_csv('dump.csv',index=False)
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> Nikhil VJ
>>>>>>> https://nikhilvj.co.in
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 6, 2022 at 10:53 AM Akshay Kumar <akshay.cg...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMD_CSV.7z
>>>>>>>> <https://drive.google.com/file/d/1kYrptA1jYDhrAE5OiNQSQQrcpncQaRo_/view?usp=drive_web>
>>>>>>>> Clipped_rain_INDIA.7z
>>>>>>>> <https://drive.google.com/file/d/1JbZmrSJGMK87Ze6X2MPRWkt_SDw2I5TN/view?usp=drive_web>
>>>>>>>> Hi
>>>>>>>> Please find the attached shapefiles of the IMD Gridded rainfall
>>>>>>>> dataset clipped for India for the years 1990 to 2021. Also, I am
>>>>>>>> attaching
>>>>>>>> the csv files of rainfall datasets, which were further used for
>>>>>>>> creating
>>>>>>>> shapefiles of corresponding years. Please let me know if it solves your
>>>>>>>> purpose.
>>>>>>>> You can clip out Himachal from the Indian shapefile.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Profound RegardsAkshay Kumar*
>>>>>>>>
>>>>>>>> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>*
>>>>>>>> *+91-8198834098*
>>>>>>>>
>>>>>>>> P Please don't print this e-mail unless you really need to. Go
>>>>>>>> Green.
>>>>>>>> *Every 3000 A4 Size papers costs 1 tree.*
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 6, 2022 at 10:39 AM georg...@gmail.com <
>>>>>>>> georgel...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thank you Nikhil, As Vastav said that link is no longer available.
>>>>>>>>> Is there any chance that you have collected the data? I want data for
>>>>>>>>> Himachal from 2000 - 2021
>>>>>>>>> Thank you
>>>>>>>>>
>>>>>>>>> On Thursday, 6 October 2022 at 03:11:55 UTC+5:30
>>>>>>>>> f201...@hyderabad.bits-pilani.ac.in wrote:
>>>>>>>>>
>>>>>>>>>> Hi Nikhil
>>>>>>>>>> It seems like the link used to get data no longer works. The
>>>>>>>>>> government has likely made the data inaccessible. Did you download
>>>>>>>>>> all the
>>>>>>>>>> files by any chance? Would it be possible to upload those in google
>>>>>>>>>> drive?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Vastav Ratra
>>>>>>>>>>
>>>>>>>>>> On Wednesday, October 5, 2022 at 3:58:10 AM UTC+2
>>>>>>>>>> nikh...@gmail.com wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Georgelio,
>>>>>>>>>>>
>>>>>>>>>>> I was able to extract the rainfall data just fine.
>>>>>>>>>>> Bit of programming is needed to get rid of junk data, and we
>>>>>>>>>>> need to work our way thru lot of files for multi-yr. Here is a repo
>>>>>>>>>>> with
>>>>>>>>>>> work done on it.
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/answerquest/IMD-grid-data-work
>>>>>>>>>>>
>>>>>>>>>>> I made a viz of the data for a location nr my home in Pune, and
>>>>>>>>>>> the figures appear to corelate with local history (Panshet flood of
>>>>>>>>>>> 1962
>>>>>>>>>>> for example shows up vividly). But I'd really like to know what the
>>>>>>>>>>> original raw data was using which the gridded data was obtained.
>>>>>>>>>>>
>>>>>>>>>>> I've loaded it all into a DB; should be able to get out specific
>>>>>>>>>>> data needed, let me know if you need for a particular location(s)
>>>>>>>>>>> and years
>>>>>>>>>>> range.
>>>>>>>>>>>
>>>>>>>>>>> Is there a use case for uploading a mirror dataset to zenodo or
>>>>>>>>>>> kaggle or someplace in simpler CSV form? (Big upload so better if
>>>>>>>>>>> there is
>>>>>>>>>>> a demand for it. Or, better if someone from the institution itself
>>>>>>>>>>> does
>>>>>>>>>>> it). I love the multi-dimensional array format they've used, but
>>>>>>>>>>> the junk
>>>>>>>>>>> values undo the storage savings (a .7z of flattened CSV may be
>>>>>>>>>>> smaller),
>>>>>>>>>>> and it's tougher to access for folks not in the know.
>>>>>>>>>>>
>>>>>>>>>>> - Nikhil
>>>>>>>>>>> https://nikhilvj.co.in
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Oct 4, 2022, 15:37 georg...@gmail.com <
>>>>>>>>>>> georg...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>> Gridded rainfall data for India in IMD is not working. I need
>>>>>>>>>>>> rainfall data for Himachal from 2010 to 2022.
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Datameet is a community of Data Science enthusiasts in India.
>>>>>>>>>>>> Know more about us by visiting http://datameet.org
>>>>>>>>>>>> ---
>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>> Google Groups "datameet" group.
>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>> it, send an email to datameet+u...@googlegroups.com.
>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/d/msgid/datameet/b6a669b9-6b30-41d6-bc22-cb960814bc6en%40googlegroups.com
>>>>>>>>>>>> <https://groups.google.com/d/msgid/datameet/b6a669b9-6b30-41d6-bc22-cb960814bc6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know
>>>>>>>>> more about us by visiting http://datameet.org
>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "datameet" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to datameet+unsubscr...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/datameet/628cda0d-26e3-4cf0-8e4e-fa7547a5d3efn%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/datameet/628cda0d-26e3-4cf0-8e4e-fa7547a5d3efn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know
>>>>>>>> more about us by visiting http://datameet.org
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "datameet" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to datameet+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/datameet/CAH9bdtHQuk_uOM%2B0%3Da%2B8pDTPgP4HeNdbU2PaNB%2B2bT8CzAPiiw%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/datameet/CAH9bdtHQuk_uOM%2B0%3Da%2B8pDTPgP4HeNdbU2PaNB%2B2bT8CzAPiiw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>> Datameet is a community of Data Science enthusiasts in India. Know
>>>>>> more about us by visiting http://datameet.org
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "datameet" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to datameet+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/datameet/CAH7jeuMv5Kx5zE7YMpNXVbxezrOkc7MqFCoKnB-hJxFy-e5eqA%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/datameet/CAH7jeuMv5Kx5zE7YMpNXVbxezrOkc7MqFCoKnB-hJxFy-e5eqA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> WITH REGARDS
>>>>> DR. PANKAJ KUMAR PANDEY
>>>>>
>>>>> --
>>>>> Datameet is a community of Data Science enthusiasts in India. Know
>>>>> more about us by visiting http://datameet.org
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "datameet" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to datameet+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/datameet/CAPppCu5Eis3_%2BXudyGCu%3Dx5VrPSePV%2B01pWoyweuT5vaH3cfLg%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/datameet/CAPppCu5Eis3_%2BXudyGCu%3Dx5VrPSePV%2B01pWoyweuT5vaH3cfLg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>>
>>>> *Profound RegardsAkshay Kumar*
>>>>
>>>> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>*
>>>> *+91-8198834098*
>>>>
>>>> P Please don't print this e-mail unless you really need to. Go Green.
>>>> *Every 3000 A4 Size papers costs 1 tree.*
>>>>
>>>> --
>>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>>> about us by visiting http://datameet.org
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "datameet" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to datameet+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/datameet/CAH9bdtF4P0DLhSrkQ30aL--4SwQhHq3h_OFypZ2ws-fy9cB2VA%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/datameet/CAH9bdtF4P0DLhSrkQ30aL--4SwQhHq3h_OFypZ2ws-fy9cB2VA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>
>>>
>>> --
>>> WITH REGARDS
>>> DR. PANKAJ KUMAR PANDEY
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CAPppCu7DjAgQZQ%2BRpG_HmB%3DXTfdCsFmo%2Bk7fe-soBXZ_t_BTEg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CAPppCu7DjAgQZQ%2BRpG_HmB%3DXTfdCsFmo%2Bk7fe-soBXZ_t_BTEg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAH7jeuOg1qUwCChb7RnhLi%2Bodf3YggfF3A7tO92jekQAjOCpzA%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAH7jeuOg1qUwCChb7RnhLi%2Bodf3YggfF3A7tO92jekQAjOCpzA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
WITH REGARDS
DR. PANKAJ KUMAR PANDEY
--
Datameet is a community of Data Science enthusiasts in India. Know more about
us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/datameet/CAPppCu66ncBcpz--PO%2Br%3DUkF2aQX6f%3DEK_PcjC8gSM95sh5HpQ%40mail.gmail.com.
