[datameet] Re: Script to download from data.gov.in
Thanks Shantanu. Registering now. Hopefully this will make data download easier. On Thursday, 5 November 2015 12:23:44 UTC+5:30, shantanu oak wrote: > > Hi, > If you register with data.gov.in - then you will get an API key that can > be used to access JSON data directly in your PHP/ Python script. > > -- Shantanu > > On Wednesday, 4 November 2015 11:25:27 UTC+5:30, Karthik Shashidhar wrote: >> >> One annoying thing about data.gov.in is that they put out data in >> multiple files in a single catalog, and with the number of such files >> occasionally being large, downloading them manually is an extremely painful >> process. >> >> Just wanted to check if any of you has written a script to download all >> CSVs from a particular catalog on data.gov.in , and if so, if you can >> share it. >> >> Cheers! >> Karthik > > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[datameet] Re: Script to download from data.gov.in
Hi,
If you know little bit of python, you can parse JSON using modules like
pandas and requests.
import json
import requests
import pandas as pd
d =
json.loads(requests.get('https://data.gov.in/node/305681/datastore/export/json').text)
pd.DataFrame(d["data"], columns=[x["label"] for x in d["fields"]])
You read the entire page and save it as a python object called "d". The
variable "d" has 2 parts - fields and data. We extract fields as column
headings and read data rows into pandas dataframe. Some other JSON output
may have different format and you will extract only "records" part of the
string dictionary object "d" like this...
pd.DataFrame([x for x in d["records"]])
You will have to read raw JSON to understand how it is formatted. This is
one time activity and then you can automate everything.
-- Shantanu
On Friday, 6 November 2015 13:40:46 UTC+5:30, Karthik Shashidhar wrote:
>
> Thanks Shantanu.
>
> Registering now. Hopefully this will make data download easier.
>
> On Thursday, 5 November 2015 12:23:44 UTC+5:30, shantanu oak wrote:
>>
>> Hi,
>> If you register with data.gov.in - then you will get an API key that can
>> be used to access JSON data directly in your PHP/ Python script.
>>
>> -- Shantanu
>>
>> On Wednesday, 4 November 2015 11:25:27 UTC+5:30, Karthik Shashidhar wrote:
>>>
>>> One annoying thing about data.gov.in is that they put out data in
>>> multiple files in a single catalog, and with the number of such files
>>> occasionally being large, downloading them manually is an extremely painful
>>> process.
>>>
>>> Just wanted to check if any of you has written a script to download all
>>> CSVs from a particular catalog on data.gov.in , and if so, if you can
>>> share it.
>>>
>>> Cheers!
>>> Karthik
>>
>>
--
Datameet is a community of Data Science enthusiasts in India. Know more about
us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[datameet] Re: Script to download from data.gov.in
Hi, If you register with data.gov.in - then you will get an API key that can be used to access JSON data directly in your PHP/ Python script. -- Shantanu On Wednesday, 4 November 2015 11:25:27 UTC+5:30, Karthik Shashidhar wrote: > > One annoying thing about data.gov.in is that they put out data in > multiple files in a single catalog, and with the number of such files > occasionally being large, downloading them manually is an extremely painful > process. > > Just wanted to check if any of you has written a script to download all > CSVs from a particular catalog on data.gov.in , and if so, if you can > share it. > > Cheers! > Karthik -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
