On Oct 25, 3:16 pm, David Jones <[email protected]> wrote: > On 24 October 2010 22:35, [email protected] > > <[email protected]> wrote: > > Ladies and Gents, > > The plan for November's meeting is to get together for a Code Dojo! This is > > an opportunity for Pythoneers and Pythonistas of all levels to flex their > > fingers and do some group coding with the aim of improving our coding > > skills. In my limited experience, no matter what your level, everyone gets > > to learn something new while having a bit of fun at the same time. > > Did you discuss _what_ you might do at a Dojo? > I'm totally open to other suggestions, but here is one I've been > thinking about a bit (and that i would like to try and turn into a > larger project, so a "starter" kit / trial would be most welcome): > > It's basically scraping weather data to make it useful for climate analysis. > > One of my hats is Clear Climate Code:http://clearclimatecode.org/. > The short version (go to the website for the longer version) is that > we have a program called ccc-gistemp that takes temperature data from > weather stations and turns it into a graph of global temperature > trends. > > The main source of data for this program is GHCN, but for various > reasons this dataset is sparse in some areas of the world where we > could do better. For example, in the UK, there are only 11 stations > with data for the last 10 years. There are hundreds of weather > stations in the UK. > > Often the data is available from websites and such, the challenge is > to get it into a form suitable for ccc-gistemp to use.
Don't know if this is useful but we have created a platform for scraping and then allowing open reuse for public data sets As this request compliments our basic idea then Scraperwiki.com would be intested to sponsor this idea with beer and pizza. We also have some example weather scrapers which could be added to if people are interested in this subject. http://scraperwiki.com/tags/weather > A concrete example that we could work on in a Dojo is this FOI > request:http://www.whatdotheyknow.com/request/weather_data_kielder_and_alice_h > > the data is in an excel file in a zip file. So ideally we would > create a Python script to: > - download the file > - unzip it > - open the excel file (xlrd, or something else?) > - reformat the data to GHCN format (which is very simple, each row has > a station identifier, a year, and 12 monthly temperature values). > > What do people think? > > (by the way, there are more places to scrape: Environment Canada make > their data available here using a very inconvenient web > form:http://www.climate.weatheroffice.gc.ca/Welcome_e.html; I've already > started scraping some of the UK Met Office data from > here:http://www.metoffice.gov.uk/climate/uk/stationdata/) > > Cheers, > drj -- To post: [email protected] To unsubscribe: [email protected] Feeds: http://groups.google.com/group/python-north-west/feeds More options: http://groups.google.com/group/python-north-west
