Hi Pankaj, Added in the webpage now:
And even after that, for some locations there just isn't data for certain years or dates in the original dataset. Those slots were filled with junk values in the .grd files like -999 - they've been sanitised out from here. You just need to keep looking for a location that gives you data and let go of the ones that don't, that is all. Not much anybody can do if IMD itself didn't archive it. Please also see the other instructions in the page. 0.5 grid : select accordingly on the map. For Assam : you'll have to select the grid points coming within assam, and try for them. For bulk work, Generate 1 link, then copy the same structures to create more links with the year value in the URL changed. -- Cheers, Nikhil VJ https://nikhilvj.co.in On Fri, Oct 14, 2022 at 9:12 PM Pankaj pandey <pkpner...@gmail.com> wrote: > Dear Nikhil I am trying to download temperature data for 2000 of Assam > and getting this massage: {"success":false,"message":"No data for give > location / year"}. What is problem Please how to download temperature data > at 0.5 degree for all grid point of assam > > On Fri, Oct 14, 2022 at 5:41 PM Nikhil VJ <nikhil...@gmail.com> wrote: > >> Hi All, >> >> Got a request regd need to simplify this IMD grid data fetching process >> for folks, and that gave me impetus to complete setting up a website where >> individual location + year data can be downloaded as simple flat CSV (which >> opens in Excel / LibreOffice Calc). >> >> So here it is: *https://server.nikhilvj.co.in/imd_data/ >> <https://server.nikhilvj.co.in/imd_data/>* >> >> There's a map and year selector to help choose, after that a link is >> generated for downloading. Open that in browser or do wget/curl >> on commandline, and it'll download a CSV with proper naming. >> >> Sample link: >> >> https://server.nikhilvj.co.in/imd_data/API/fetchData?year=2010&lat=25.25&lon=76.25&which=all >> >> You are *encouraged to scrape* : use the format above to compose your >> own link. (Hint: Excel formulas like: =A2&B2&"?year="&C2 are perfect for >> this) >> >> With a caveat: Given that it's not some fancy server that can take huge >> loads, there's a cruel and harsh rate limiter in place which will allow >> only 1 hit per 10 secs per IP address for anonymous users. >> >> If you want to bypass that, there's an email-otp login process given >> which will add a token and let you do 5 hits per second, and that won't be >> restricted by IP address. >> This would be useful in institutions where multiple users on the same >> network may want to fetch data at the same time. >> ex: a school/college hands-on session in climate data visualization where >> the host as usual forgets to pre-download the data >> So you can scrape and I can keep my server from crashing - both sides win. >> >> Source code is at https://github.com/answerquest/IMD-grid-data-work (see >> imd_data_api folder) >> >> >> -- >> Cheers, >> Nikhil VJ >> https://nikhilvj.co.in >> >> >> On Wed, Oct 12, 2022 at 8:30 AM Nikhil VJ <nikhil...@gmail.com> wrote: >> >>> Hi Pankaj, >>> >>> The place initially referred to is archival data - by definition it's >>> historic and not realtime. >>> What I do see on https://imdpune.gov.in/lrfindex.php is "Gridded Data >>> Real Time" -> you might want to explore that. >>> Over there, Daily 0.5x0.5 grid temp data links are there. >>> You can select a single date there and download the file. >>> >>> I don't know right now how to open it, though. imdlib library that I've >>> used - I didn't see anything about daily data. >>> Here's its documentation: >>> https://imdlib.readthedocs.io/en/latest/Usage.html >>> >>> I've posted a query about it in the lib's github : >>> https://github.com/iamsaswata/imdlib/issues/17 , let's see if the >>> author responds. Perhaps some inspection of the source code also might >>> yield a way to open the daily data. >>> >>> PS: Next time do initiate a separate conversation with a different >>> subject line - as here you have asked about temperature data but the >>> subject reads rainfall data. Becomes less discoverable over the open >>> internet or after a long period of time. >>> >>> -- >>> Cheers, >>> Nikhil VJ >>> https://nikhilvj.co.in >>> >>> >>> On Thu, Oct 6, 2022 at 11:56 PM Pankaj pandey <pkpner...@gmail.com> >>> wrote: >>> >>>> Thanks Akshay for sharing the code. Actually I want to download tmax >>>> and tmin data at 0.5 degree not at 1 degree for 2000 to 2021. As I want to >>>> dowanload gridded data real time Download Daily Maximum Temperature >>>> data ( 0.5 x 0.5 degree) Binary File (Real-time). In this page only >>>> the select data option is available. Should i replace year with data in >>>> code. >>>> It is requested please modify code for this case >>>> >>>> with regards >>>> p.k.pANDEY >>>> >>>> >>>> >>>> On Thu, Oct 6, 2022 at 11:30 PM Akshay Kumar <akshay.cg...@gmail.com> >>>> wrote: >>>> >>>>> Dear Sir >>>>> Pl check this link. >>>>> >>>>> https://github.com/Manisht9/imdData >>>>> >>>>> >>>>> On Thu, 6 Oct 2022 at 10:55 PM, Pankaj pandey <pkpner...@gmail.com> >>>>> wrote: >>>>> >>>>>> Dear Nikhil how can batch download data, >>>>>> https://imdpune.gov.in/lrfindex.php. This page give data wise data. >>>>>> how can download suppose 20 year data of temperature. Any python code to >>>>>> do this to bulk download in csv. Please update any code will very much >>>>>> neede >>>>>> >>>>>> With Regards >>>>>> P.K.Pandey >>>>>> >>>>>> >>>>>> On Thu, Oct 6, 2022 at 7:28 PM Nikhil VJ <nikhil...@gmail.com> wrote: >>>>>> >>>>>>> Hi Georgelio, Vastav, >>>>>>> >>>>>>> While this link is down: >>>>>>> https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html >>>>>>> >>>>>>> Upon browsing around the IMD site I found under Climate Monitoring >>>>>>> in top menu, https://imdpune.gov.in/lrfindex.php >>>>>>> Side menu: Gridded Data Archive -> All the data is there and >>>>>>> available just fine. >>>>>>> >>>>>>> >>>>>>> So, taking offline my data dump to save my server's bandwidth. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Cheers, >>>>>>> Nikhil VJ >>>>>>> https://nikhilvj.co.in >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 6, 2022 at 6:38 PM Nikhil VJ <nikhil...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Wow, Dead Internet Theory vibes 🤐 >>>>>>>> >>>>>>>> Ok, I've moved all the gridded data file copies that I had at my >>>>>>>> end to: https://server.nikhilvj.co.in/dump/imd_grid/ >>>>>>>> >>>>>>>> Disclaimer: they were download from <currently down> source >>>>>>>> <https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html> >>>>>>>> on 13th July 2022, and their sha256 checksums are at: >>>>>>>> >>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_rain.txt >>>>>>>> >>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmax.txt >>>>>>>> >>>>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmin.txt >>>>>>>> >>>>>>>> Beyond that digital trail, no guarantee on integrity of the data >>>>>>>> from my side. Use at own risk. >>>>>>>> >>>>>>>> This code snippet should help you turn a file into a flat CSV table: >>>>>>>> >>>>>>>> import imdlib >>>>>>>> rain1 = imdlib.open_data('rain', 2010, 2010, >>>>>>>> 'yearwise').get_xarray().to_dataframe() >>>>>>>> rain2 = rain1[rain1['rain'] > -100].reset_index() >>>>>>>> rain2.to_csv('dump.csv',index=False) >>>>>>>> >>>>>>>> -- >>>>>>>> Cheers, >>>>>>>> Nikhil VJ >>>>>>>> https://nikhilvj.co.in >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 6, 2022 at 10:53 AM Akshay Kumar < >>>>>>>> akshay.cg...@gmail.com> wrote: >>>>>>>> >>>>>>>>> IMD_CSV.7z >>>>>>>>> <https://drive.google.com/file/d/1kYrptA1jYDhrAE5OiNQSQQrcpncQaRo_/view?usp=drive_web> >>>>>>>>> Clipped_rain_INDIA.7z >>>>>>>>> <https://drive.google.com/file/d/1JbZmrSJGMK87Ze6X2MPRWkt_SDw2I5TN/view?usp=drive_web> >>>>>>>>> Hi >>>>>>>>> Please find the attached shapefiles of the IMD Gridded rainfall >>>>>>>>> dataset clipped for India for the years 1990 to 2021. Also, I am >>>>>>>>> attaching >>>>>>>>> the csv files of rainfall datasets, which were further used for >>>>>>>>> creating >>>>>>>>> shapefiles of corresponding years. Please let me know if it solves >>>>>>>>> your >>>>>>>>> purpose. >>>>>>>>> You can clip out Himachal from the Indian shapefile. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Profound RegardsAkshay Kumar* >>>>>>>>> >>>>>>>>> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>* >>>>>>>>> *+91-8198834098* >>>>>>>>> >>>>>>>>> P Please don't print this e-mail unless you really need to. Go >>>>>>>>> Green. >>>>>>>>> *Every 3000 A4 Size papers costs 1 tree.* >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 6, 2022 at 10:39 AM georg...@gmail.com < >>>>>>>>> georgel...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thank you Nikhil, As Vastav said that link is no longer >>>>>>>>>> available. Is there any chance that you have collected the data? I >>>>>>>>>> want >>>>>>>>>> data for Himachal from 2000 - 2021 >>>>>>>>>> Thank you >>>>>>>>>> >>>>>>>>>> On Thursday, 6 October 2022 at 03:11:55 UTC+5:30 >>>>>>>>>> f201...@hyderabad.bits-pilani.ac.in wrote: >>>>>>>>>> >>>>>>>>>>> Hi Nikhil >>>>>>>>>>> It seems like the link used to get data no longer works. The >>>>>>>>>>> government has likely made the data inaccessible. Did you download >>>>>>>>>>> all the >>>>>>>>>>> files by any chance? Would it be possible to upload those in google >>>>>>>>>>> drive? >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Vastav Ratra >>>>>>>>>>> >>>>>>>>>>> On Wednesday, October 5, 2022 at 3:58:10 AM UTC+2 >>>>>>>>>>> nikh...@gmail.com wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Georgelio, >>>>>>>>>>>> >>>>>>>>>>>> I was able to extract the rainfall data just fine. >>>>>>>>>>>> Bit of programming is needed to get rid of junk data, and we >>>>>>>>>>>> need to work our way thru lot of files for multi-yr. Here is a >>>>>>>>>>>> repo with >>>>>>>>>>>> work done on it. >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/answerquest/IMD-grid-data-work >>>>>>>>>>>> >>>>>>>>>>>> I made a viz of the data for a location nr my home in Pune, and >>>>>>>>>>>> the figures appear to corelate with local history (Panshet flood >>>>>>>>>>>> of 1962 >>>>>>>>>>>> for example shows up vividly). But I'd really like to know what the >>>>>>>>>>>> original raw data was using which the gridded data was obtained. >>>>>>>>>>>> >>>>>>>>>>>> I've loaded it all into a DB; should be able to get out >>>>>>>>>>>> specific data needed, let me know if you need for a particular >>>>>>>>>>>> location(s) >>>>>>>>>>>> and years range. >>>>>>>>>>>> >>>>>>>>>>>> Is there a use case for uploading a mirror dataset to zenodo or >>>>>>>>>>>> kaggle or someplace in simpler CSV form? (Big upload so better if >>>>>>>>>>>> there is >>>>>>>>>>>> a demand for it. Or, better if someone from the institution itself >>>>>>>>>>>> does >>>>>>>>>>>> it). I love the multi-dimensional array format they've used, but >>>>>>>>>>>> the junk >>>>>>>>>>>> values undo the storage savings (a .7z of flattened CSV may be >>>>>>>>>>>> smaller), >>>>>>>>>>>> and it's tougher to access for folks not in the know. >>>>>>>>>>>> >>>>>>>>>>>> - Nikhil >>>>>>>>>>>> https://nikhilvj.co.in >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 4, 2022, 15:37 georg...@gmail.com < >>>>>>>>>>>> georg...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> Gridded rainfall data for India in IMD is not working. I need >>>>>>>>>>>>> rainfall data for Himachal from 2010 to 2022. >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Datameet is a community of Data Science enthusiasts in India. >>>>>>>>>>>>> Know more about us by visiting http://datameet.org >>>>>>>>>>>>> --- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "datameet" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to datameet+u...@googlegroups.com. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/datameet/b6a669b9-6b30-41d6-bc22-cb960814bc6en%40googlegroups.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/datameet/b6a669b9-6b30-41d6-bc22-cb960814bc6en%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>> Datameet is a community of Data Science enthusiasts in India. >>>>>>>>>> Know more about us by visiting http://datameet.org >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "datameet" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to datameet+unsubscr...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/datameet/628cda0d-26e3-4cf0-8e4e-fa7547a5d3efn%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/datameet/628cda0d-26e3-4cf0-8e4e-fa7547a5d3efn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>>> more about us by visiting http://datameet.org >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "datameet" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to datameet+unsubscr...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/datameet/CAH9bdtHQuk_uOM%2B0%3Da%2B8pDTPgP4HeNdbU2PaNB%2B2bT8CzAPiiw%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/datameet/CAH9bdtHQuk_uOM%2B0%3Da%2B8pDTPgP4HeNdbU2PaNB%2B2bT8CzAPiiw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>> more about us by visiting http://datameet.org >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "datameet" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to datameet+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/datameet/CAH7jeuMv5Kx5zE7YMpNXVbxezrOkc7MqFCoKnB-hJxFy-e5eqA%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/datameet/CAH7jeuMv5Kx5zE7YMpNXVbxezrOkc7MqFCoKnB-hJxFy-e5eqA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> WITH REGARDS >>>>>> DR. PANKAJ KUMAR PANDEY >>>>>> >>>>>> -- >>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>> more about us by visiting http://datameet.org >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "datameet" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to datameet+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/datameet/CAPppCu5Eis3_%2BXudyGCu%3Dx5VrPSePV%2B01pWoyweuT5vaH3cfLg%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/datameet/CAPppCu5Eis3_%2BXudyGCu%3Dx5VrPSePV%2B01pWoyweuT5vaH3cfLg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> >>>>> *Profound RegardsAkshay Kumar* >>>>> >>>>> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>* >>>>> *+91-8198834098* >>>>> >>>>> P Please don't print this e-mail unless you really need to. Go Green. >>>>> *Every 3000 A4 Size papers costs 1 tree.* >>>>> >>>>> -- >>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>> more about us by visiting http://datameet.org >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "datameet" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to datameet+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/datameet/CAH9bdtF4P0DLhSrkQ30aL--4SwQhHq3h_OFypZ2ws-fy9cB2VA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/datameet/CAH9bdtF4P0DLhSrkQ30aL--4SwQhHq3h_OFypZ2ws-fy9cB2VA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> WITH REGARDS >>>> DR. PANKAJ KUMAR PANDEY >>>> >>>> -- >>>> Datameet is a community of Data Science enthusiasts in India. Know more >>>> about us by visiting http://datameet.org >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "datameet" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to datameet+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/datameet/CAPppCu7DjAgQZQ%2BRpG_HmB%3DXTfdCsFmo%2Bk7fe-soBXZ_t_BTEg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/datameet/CAPppCu7DjAgQZQ%2BRpG_HmB%3DXTfdCsFmo%2Bk7fe-soBXZ_t_BTEg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> Datameet is a community of Data Science enthusiasts in India. Know more >> about us by visiting http://datameet.org >> --- >> You received this message because you are subscribed to the Google Groups >> "datameet" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to datameet+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/datameet/CAH7jeuOg1qUwCChb7RnhLi%2Bodf3YggfF3A7tO92jekQAjOCpzA%40mail.gmail.com >> <https://groups.google.com/d/msgid/datameet/CAH7jeuOg1qUwCChb7RnhLi%2Bodf3YggfF3A7tO92jekQAjOCpzA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > WITH REGARDS > DR. PANKAJ KUMAR PANDEY > > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to datameet+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/CAPppCu66ncBcpz--PO%2Br%3DUkF2aQX6f%3DEK_PcjC8gSM95sh5HpQ%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CAPppCu66ncBcpz--PO%2Br%3DUkF2aQX6f%3DEK_PcjC8gSM95sh5HpQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuM4Zx7S5v57T1zXy9HZt7XXxK5_rBOCo6wErP394cvT0w%40mail.gmail.com.
