On 2 November 2010 19:26, David Jones <[email protected]> wrote: > On 29 October 2010 10:53, Julian Todd <[email protected]> wrote: > > I am of the view that there has to be a platform continuum between > > writing code to do a proper scraper and parse, and simply rekeying the > > data. What this means is if you happen to have a small data-set that > > you think would take a three hours to write a parser for and which is > > updated once a month, it may be more reasonable to cut-and-paste the > > data (or type it in), and then code up some some simple monitoring > > mechanism that sends out an email when the next page is uploaded. > > Agreed. Somewhere on that continuum would be software (like the > oldweather.org stuff) that assists in hand rekeying. Allowing very > easy zooming in, jumping to the next entry in the table, and so on. > > This paper http://www.springerlink.com/content/p58741881444nu11/ > contains a long record (since 1819) for Minnesota, but I can't find it > in digital form anywhere. > > So it looks like there's considerable interest in doing some group coding around climate data. Perhaps it's time to crystallise our thoughts and give some direction to the evening's event. It'll also give the eager amongst us to brush up beforehand. Here's a first bash as a climate data uninitiate:
What to code: We've talked about parsing and transforming freely available climate data into a format usable by ccc-gistemp. Can we have a pointer to the required data format? Data sources: We've had suggestions as to the source of climate data to parse and transform, including www.whatdotheyknow.com, www.oldweather.org and a paper from www.springerlink.com. Between David and Julian, I'm guessing we'll have more than enough data to chew on. What to code in: Python of course! And ScraperWiki (http://scraperwiki.com/) specifically? Do we envisage using ScraperWiki exclusively? Do we have ScraperWiki experts out there who plan to attend? Tools: We need people to bring their laptops! On the basis that we'll be pair programming, the magic ratio will be one laptop for every two attendees. I'm hoping either or both of David or Julian would be happy to give direction at the event itself. You might like to get the show on the road by giving a quick intro at the start on climate data (and possibly ScraperWiki) ... :) Would you consider it useful to create a simple crib sheet beforehand outlining, for example, data sources | data formats | what the data is | what we're aiming to do etc., and probably most importantly, point us to a simple example? Cheers, Safe > drj > > -- > To post: [email protected] > To unsubscribe: [email protected] > Feeds: http://groups.google.com/group/python-north-west/feeds > More options: http://groups.google.com/group/python-north-west > -- To post: [email protected] To unsubscribe: [email protected] Feeds: http://groups.google.com/group/python-north-west/feeds More options: http://groups.google.com/group/python-north-west
