Re: [datameet] Data of Indian Railways

[Nikhil VJ]

Heads-up for folks in Pune : Mark your calendars for Jan 12th, DMers are 
organizing a meetup to get together and work on this data. We'll post 
details here when finalized.

---
Hi Arun,

Good is great, and I tell this to my interns : Never chase perfection, 
target 80% and take a break; then shoot for 80% in the next round and so on 
and we'll reach perfection iteratively. ;)

That explains why the site didn't seem to be fetching any lat-long data.. 
it was all in that js. 
I still can't make out how the map is loading the tracks, though.

--
More on the train (ie, route) info:
Two API calls are made my TrainOnMap site to fetch route info:

Taking example of train no. 12438:

1. NTES : 
https://enquiry.indianrail.gov.in/ntes/NTES?action=getTrainData=12438
>From this we get the following data:
- train (route) code, name
- schedule with each stop code, arrival, departure times. But this is main 
stops only.
- but lat-long of station codes not given.
- validity dates (at least from which date this schedule is active)
- days of the week it runs on (as a 7 digit binary num like 1001010 )
- train type
- journey duration
- latest journey actual times, delays, timestamp of last update

2. RailGeo: 
https://enquiry.indianrail.gov.in/ntes/RailGeo?action=getTrainSchedule=12438=6%20Jan%202019
- This gives a flat json array (equivalent to flat table) of 205 rows,
- many station codes. Presumably, these are all the stations the train 
passes, not just the main ones. Hence, intermediate stops can be obtained 
from here.
- but no station names or lat-longs (That "Geo" makes the heart skip a beat 
but alas, no dice.)
- other timing info is all zero'd so N/A.
- day-count tells if it's the next day.
- the date parameter in the URL can presumably be taken from the response 
of previous query


---

Hi Jasvinder, 
Glad you liked it! We got some students together and mapped around 60 
routes, 3000+ stops this past weekend with a next version of that. I'll 
start work on a version for railway station data.
By android I assume you mean mobile-friendly. Been trying for that, but its 
a bit difficult to re-arrange the various parts and make all the 
functionality work. I really doubt if the precision needed can be obtained 
without a mouse. But you're welcome to have a go at it, we can talk it over.


Regards
Nikhil VJ
Pune, India


On Monday, January 7, 2019 at 10:05:03 AM UTC+5:30, Jasvinder Singh wrote:
>
> Dear Nikhil,
>
> Saw the interface. Its quite simple and easy to use even by newbies. 
> However is an android version of the same available?
> It would be easy to update the databank whenever one is traveling and 
> comes across an unfamiliar station.
>
> Regards,
>
> Jasvinder Singh
>
>
> On Fri, Jan 4, 2019 at 3:51 PM Nikhil VJ > 
> wrote:
>
>> Hi, 
>>
>> If there's interest I can set up a mapping interface to crowdsource 
>> lat-longs. Along the lines of this: https://fuzzymapper.herokuapp.com/ 
>> But I would take a week to set up so tell. Though it would be best to get 
>> existing lat-long sets with the official station code and import them in, 
>> this can help cover the laggards.
>>
>>
>> Also, if collective work on OpenRefine is required then I can help set it 
>> up on cloud and keep protected edit access. That won't take more time.
>>
>>
>> Regards
>> Nikhil VJ, Pune
>>
>>
>> On Friday, January 4, 2019 at 12:43:17 PM UTC+5:30, Jasvinder Singh wrote:
>>>
>>> Dear Arun,
>>> Exactly the type of simple data sheet that newbies can understand. 
>>> However how location (Coordinates) is linked in this file?
>>>
>>> On Fri, Jan 4, 2019 at 12:11 PM Arun Ganesh  wrote:
>>>
 Spreadsheet if anyone wants to explore: 
 https://docs.google.com/spreadsheets/d/1AFwl_5cB9qD39VWNox1LoeL3tGaGB22f7p4vc7IyMqY/edit#gid=0

 There are 16,770 station entries of which 11,660 seem to be currently 
 operational according to the expiry date of 2999. 

 Filtering out goods stations, there are 9835 entries. This still seems 
 to include a few yards and cabins that are not legitimate stations. Also 
 noticed quite a few spelling and formatting issues in the names. The 
 station codes look correct. Some amount of manual cleanup is needed on 
 this 
 list. 

 The official number of stations according to IR is 7349 stations (as 
 of 2017) 
 
 and 1817 halts/block huts (2013) 
 .
  
  

 On Fri, Jan 4, 2019 at 11:53 AM Arun Ganesh  wrote:

> Beauty of the internet, crawler got done by Srihari: 
> https://twitter.com/sriharithalla/status/1080801313707896837
>
> JSON data: 
> https://api.apify.com/v1/execs/TsBwnYutP5u9FCKp5/results?format=json=1
>
> I'm in the process of 

Re: [datameet] Data of Indian Railways

[Jasvinder Singh]

Dear Nikhil,

Saw the interface. Its quite simple and easy to use even by newbies.
However is an android version of the same available?
It would be easy to update the databank whenever one is traveling and comes
across an unfamiliar station.

Regards,

Jasvinder Singh


On Fri, Jan 4, 2019 at 3:51 PM Nikhil VJ  wrote:

> Hi,
>
> If there's interest I can set up a mapping interface to crowdsource
> lat-longs. Along the lines of this: https://fuzzymapper.herokuapp.com/
> But I would take a week to set up so tell. Though it would be best to get
> existing lat-long sets with the official station code and import them in,
> this can help cover the laggards.
>
>
> Also, if collective work on OpenRefine is required then I can help set it
> up on cloud and keep protected edit access. That won't take more time.
>
>
> Regards
> Nikhil VJ, Pune
>
>
> On Friday, January 4, 2019 at 12:43:17 PM UTC+5:30, Jasvinder Singh wrote:
>>
>> Dear Arun,
>> Exactly the type of simple data sheet that newbies can understand.
>> However how location (Coordinates) is linked in this file?
>>
>> On Fri, Jan 4, 2019 at 12:11 PM Arun Ganesh  wrote:
>>
>>> Spreadsheet if anyone wants to explore:
>>> https://docs.google.com/spreadsheets/d/1AFwl_5cB9qD39VWNox1LoeL3tGaGB22f7p4vc7IyMqY/edit#gid=0
>>>
>>> There are 16,770 station entries of which 11,660 seem to be currently
>>> operational according to the expiry date of 2999.
>>>
>>> Filtering out goods stations, there are 9835 entries. This still seems
>>> to include a few yards and cabins that are not legitimate stations. Also
>>> noticed quite a few spelling and formatting issues in the names. The
>>> station codes look correct. Some amount of manual cleanup is needed on this
>>> list.
>>>
>>> The official number of stations according to IR is 7349 stations (as of
>>> 2017)
>>> 
>>> and 1817 halts/block huts (2013)
>>> .
>>>
>>>
>>> On Fri, Jan 4, 2019 at 11:53 AM Arun Ganesh  wrote:
>>>
 Beauty of the internet, crawler got done by Srihari:
 https://twitter.com/sriharithalla/status/1080801313707896837

 JSON data:
 https://api.apify.com/v1/execs/TsBwnYutP5u9FCKp5/results?format=json=1

 I'm in the process of doing a little bit of cleanup using openrefine
 and will share on a spreadsheet.

 On Fri, Jan 4, 2019 at 10:01 AM Jasvinder Singh 
 wrote:

> Dear All,
>
> Not all the members are familiar with intricacies of the data
> collection for such projects. Since this seems to be a crowd sourcing
> endeavour, I suggest that the basic data collection protocol be enumerated
> for newbies so that they can also contribute data which can then be put in
> proper format by professionals.
>
> Regards,
>
> Jasvinder Singh
>
> On Tue, Dec 25, 2018 at 10:32 AM Nikhil VJ  wrote:
>
>> Hi folks,
>>
>> There's a project afoot in the OpenStreetMap and Wikidata communities
>> to get together Indian Railways data.
>>
>> One major part of it: Properly mapping all the railway stations of
>> India, and ensuring they have wikidata entries.
>>
>> Here's a wiki page set up for it:
>> https://www.wikidata.org/wiki/Wikidata:WikiProject_Indian_Railways
>>
>> I'm cross-posting from OpenStreetMap India Telegram
>>  group:
>>
>> (Arun Ganesh): There seems to be around 7000 stations located. There
>> still ~1.5k missing. A lot more need names, refs and wikidata links.
>> Overpass: *http://overpass-turbo.eu/s/EC4
>> *
>>
>>
>> (Srihari Thalla) : Last year I created two MapRoulette Challenges to
>> tag station codes and add Wiki tags
>> *https://maproulette.org/mr3/challenge/2403
>> *
>> *https://maproulette.org/mr3/challenge/2404
>> *
>>
>> 
>>
>> The overpass query above queries the whole country and may be slow or
>> timeout. I adapted the query to work only on the map area being
>> viewed, so you can zoom into smaller regions. And changed a few things,
>> included a legend in the comments to explain.
>>
>> *https://overpass-turbo.eu/s/EL9 *
>>
>>
>> Want to get involved? Engage here.
>>
>>
>>
>> Regards
>>
>> Nikhil VJ
>>
>> Pune, India
>>
>>
>> PS: Posting on an older thread from '15 that had the perfect subject
>> line - didn't want to create yet another new thread. Pro tip: Use 
>> Datameet from
>> google groups  -
>> its more fun and you can 

Re: [datameet] Data of Indian Railways

[Arun Ganesh]

On Sun, Jan 6, 2019 at 11:19 AM Nikhil VJ  wrote:

> Hi,
>
> We might be able to get some data from Train On Map site of Indian
> Railways?
>
> https://enquiry.indianrail.gov.in/ntes/trainOnMap.jsp
>
> (good to see the map is working again.. last few months it had been down
> after google api changes)
>
>
> The browser console is yielding quite some stuff.
>
> One JS file there is holding some data:
>
> https://enquiry.indianrail.gov.in/ntes/js/stnCodesWithNamesArrayStr.js
>
>
This is amazing, wrote a simple script and converted the station list into
a CSV:
https://docs.google.com/spreadsheets/d/1AFwl_5cB9qD39VWNox1LoeL3tGaGB22f7p4vc7IyMqY/edit?usp=sharing

There are 10,482 entries and 9,240 have a coordinate. The list includes
many block cabins and freight sidings which are not stations.

Did a quick quality check, Yesvantpur is Yasvantpur and KSR Bengaluru has
the old name in Hindi बेंगलोर सिटी जं.

Overall quite good, but sadly not perfect.

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[datameet] Re: 2018 India Village GIS Data Project

[Naraina Damle]

Hi Dhaval, 

Actually I keep doing things in an Intermittent manner. 
Then I move to something else. 

However I keep all my data safe. 

Here is the link to the Datameet folder on My Google Drive. 

There could be other things at other places. But Gujarat Data is here. 

https://drive.google.com/drive/folders/0Bw9Ef2_V5qDeOTFwV1U1WDA5Rnc?usp=sharing






On Friday, January 4, 2019 at 6:35:59 PM UTC+5:30, Dhaval Shukla wrote:
>
> Hey
>
> Naraina had done something with Gujarat district and village shape files 
> some time ago, but for the life of me I am not able to find it on this 
> group today. I downloaded it at the time about a week ago.
>
> This village and district data hasn't been uploaded to the datameet github 
> nor yours.
>
> Naraina, I wanted to know if there is a reason why. It is most accurate I 
> have found of Gujarat till date (although there are some errors). 
>
>
>
>
> On Wednesday, 21 March 2018 20:04:16 UTC+5:30, Justin wrote:
>>
>> I plan on trying to update individual states to 2018 standards/ amounts/ 
>> etc. I know this is a crazy idea, but I am going to start.
>>
>> I have started with Goa, it has been a challenge. I thought I was going 
>> to finish over two weeks ago, but just haven't had time and have been doing 
>> more tracing of villages that what I thought I was going to have to do.
>>
>> I am trying to align amounts with http://lgdirectory.gov.in/ and 
>> http://nlrmp.nic.in/faces/masterReport/masterReport.xhtml
>>
>> Sometimes these may disagree and may be missing towns, but I am trying to 
>> make sure I account for those somehow. Once I finish, I will upload hoping 
>> for feedback as per what I did wrong, am missing, or anything else.
>>
>> I think a state-by-state way is the only way to successfully do this. 
>> Trying to do all 6-8 lakh would be a nightmare, so breaking it down to 
>> several hundred - a few thousand would be easiest and manageable.
>>
>> Please let me know if you would like to help, start mapping a state, or 
>> give any feedback, ideas, data, solutions, etc.
>>
>> Thank you all so much!
>>
>> Justin
>>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[datameet] Require suggestions about services to be hosted for a state gis portal.

[Sandeep Kumar]

Hello everyone,

I have been a member of this group and now i got a chance  to work on 
building SDI portal for Andhra Pradesh. I have currently created a portal 
for data discovery https://apsac.ap.gov.in/geonetwork/ more data to follow 
in the same portal, also we do have an ARCGIS portal 
https://apsac.ap.gov.in/gisportal/home/ . I want suggestions how i can make 
data easily accessible for public and other stakeholders, you can suggest 
me any open software or a use case that can benefit people working in the 
state in field of GIS.
 

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.