Hi Abhishek, Right-click on page -> Inspect -> opens the browser console -> Go to Network tab , then browse around the site and check out the comings and goings between site and server.
I was able to get the data we're seeing in the website coming in my command prompt (which means: one can capture this) with this simplified cURL command: curl 'https://nwp.imd.gov.in/blf/blf_temp/block.php' --compressed --data-raw 'dis=22AMRAVATI' How to get these district codes: inspecting one page before.. curl --compressed ' https://nwp.imd.gov.in/blf/blf_temp/dis.php?value=22maharashtra' How to get the state codes: just scrape from the html of the page you shared: https://nwp.imd.gov.in/blf/blf_temp/ Before my foray into python, I used to use notepad++ and libreoffice Calc (their raw text import dialog leaves excel in the dust) to separate out the data I needed from the html tags etc. I'd use spreadsheet formulas to even generate command-prompt commands in bulk. Fun times. For limited jobs, there's things you can do there in a few mins which would take hours of coding. Maybe you can set these up in Octoparse (i have no experience with that as I just roll my own code) or some other tools to get the data you need. All the best! -- Cheers, Nikhil VJ https://nikhilvj.co.in On Thu, May 27, 2021 at 3:32 AM [email protected] <[email protected]> wrote: > Hi, > > https://nwp.imd.gov.in/blf/blf_temp/ > > In the link above, we can see the 5 days weather prediction by districts - > however, there are a lot of clicks that are required to reach any one > particular district. Please help in identifying a method to collate all > India Rainfall data in a single table. > > Tried to use Octoparse - however, either I am using it incorrectly or else > it doesn't work in this site. any other source of the same data is also > welcome. > > regards, > Abhishek > > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com > <https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuM80boH2pdNrOkef3wQcTzFD9Yg0LVJaQL-SNr8pU_GbA%40mail.gmail.com.
