Hi, (replying to this thread <https://groups.google.com/g/datameet/c/FrLXpbpRnHg/m/FWJgNL5ZAQAJ> with a changed subject line) I searched on net for how to extract the imdb gridded data (Sai Kirshna thanks for ur program too) And I found this video: https://www.youtube.com/watch?v=40Dvj6MwGTs by OpenGeoLab <https://www.youtube.com/c/OpenGeoLab> channel on youtube (done some pretty good training work in GIS!)
Mentions: "imdblib <https://imdlib.readthedocs.io/en/latest/>" python package.. and it looks like that one takes care of a lot of the complexity for us! Big thanks to Saswati Nandi <https://github.com/iamsaswata> who authored it. With a bit of looking up <https://docs.xarray.dev/en/stable/user-guide/pandas.html#dataset-and-dataframe> I was able to bring the data out to a simple flat table format that we all know and love. Sharing the code of a sample extraction here: https://github.com/answerquest/IMD-grid-data-work/blob/main/2022-07-13%20IMD%20Gridded%20data%20extraction%20sample%20flow.ipynb I took out some data for nearby Pune, and have done a quick viz in this google doc: https://docs.google.com/spreadsheets/d/13j3kMvGdr52_dX-4nG9YDFCu3ZO_sYadFmYeiZJVgWo/edit?usp=sharing *Observations:* 1. There is a lot of junk data in there : for 1950s yrs that I saw it was over 60%. Fixed values like -999 are filled in where there wasn't any reading for that particular place and date - probably because the data system being used couldn't work with nulls. So it's important to get rid of the junk data points before moving forward. 2. If I 7z-zipped the flat table csv after removing the junk data, and whadyaknow, the result is smaller. (tip for folks managing this department in IMD. Also, check out HDF5 format) 3. There's 3 data items: Max temp, min temp, rainfall. 4. Temp data is resolved to .5 lat-long points and available from 1950. Rain data is resolved to .25 lat-long points and available from 1901. *Next steps:* As usual I'm going to setup a week-long script (going year by year only, with plenty of pauses so that it doesn't overload the IMD site, mind!), extract it all and load it up into a Postgresql DB / API etc for accessing. *Would anyone out there like to collaborate on a visualization?* We have data that is in point lat-long form, and is temporal with date-wise resolution. So, map-based combined with time series and/or animation would be good. -- Cheers, Nikhil VJ https://nikhilvj.co.in On Fri, Jul 8, 2022 at 8:00 AM Dammalapati Sai Krishna < [email protected]> wrote: > The method to acquire this data and to aggregate it at district levels can > be taken from here: > https://github.com/d-saikrishna/GIS_RemoteSensing/blob/master/Covid_Env/Data%20Collection%20and%20Pre-Processing.ipynb > > Let me know if you need more help. > > Sai Krishna > > On 08-Jul-2022, at 12:53 AM, Jayshree Dutta <[email protected]> wrote: > > > Hello, I'm doing my Ph.D in Chennai and currently visiting researcher in > Austria. I need India's district wise data on Temperature (avg, or min > -max), humidity, wind velocity and total sun shine hours. Can anyone help > me getting these datasets? Atleast any of them? Anytime after 2015 sound be > fine, and of course more recent the better! > > With regards > Jay > > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/CAFFWLbCs3vAtVmdp1u6yfjhp3Rwvod0N6axt5_m_o-ONWWUJuQ%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CAFFWLbCs3vAtVmdp1u6yfjhp3Rwvod0N6axt5_m_o-ONWWUJuQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/BDDF6844-ECF4-4B08-A048-BAC7F1F549AC%40gmail.com > <https://groups.google.com/d/msgid/datameet/BDDF6844-ECF4-4B08-A048-BAC7F1F549AC%40gmail.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuNH8w9tTSFofcvrzibAFTLJM2VBz64ELk1tPxzgn_F5Yw%40mail.gmail.com.
