Paul Check out the code at the github repository. The db schema (rubyfied version unfortunately, but very readable) is at
http://github.com/CountCulture/twfy_local_parser/blob/b3354949a257853910f87d6a0888ac6159a37ebb/db/schema.rb Actually, Brighton & Hove uses Modern.gov as its CMS for councillor/committee data, which already has a scraper/parser written, so it just took me a couple of minutes to populate the data for that (http://www.theyworkforyoulocal.com/councils/45). Maybe you could cast your eye over it to see if there are any probs (I'm sure there are). Cheers C paul perrin wrote: > Hi > > I saw your posting on mysociety. I sometimes have some spare time and > might be able to help out with brighton and hove (my local council). I > am really a software engineer in .NET, SQLServer (and XML, XSLT) type > bod, not ruby, so maybe not so useful... > > But if your scraping is table/data driven I could pick out the paths > needed for the data. Have you done a data model for the info (I am > pretty hot on data models). > > Regards > > Paul Perrin > /)/+) > > 2009/6/16 CountCulture <[email protected] > <mailto:[email protected]>> > > Quick note about something I've been working on in my spare time: > > http://theyworkforyoulocal.com -- a small app to scrape and parse > local > authority info. > > At the moment, it's barely more than a proof of concept, with only > about > 20 or so councils parsed, and even then only current councillors, > committees, committee membership and forthcoming meetings are parsed. > > On the upside, it's fairly quick for me to add new parsers for > councils > (and reuse ones already written if they use same CMS), there's an API > built in (basically just add .json or .xml to get the info as json or > XML), and there's lots of potential. > > Getting this far has also been an education in understanding what a > full-blown twfy_local might look like (in general there seems no > way to > see how councillors voted, for example), the need for such a resource > (there's no publicly available central repository for council election > results, for example), and the sorry state of local authority websites > (just finding a list of councillors is a challenge on some, and don't > get me started on the HTML markup). > > Comments welcome. Code is at > http://github.com/CountCulture/twfy_local_parser/ (I'll probably > GPL it > soon). Bug reports at > http://github.com/CountCulture/twfy_local_parser/issues and offers of > help to countculture at googlemail dot com. > > I'd especially be interested in hearing from anyone who's got any > knowledge about local authority CMSs (e.g. there seem to be several > different versions of Modern.Gov producing different URLs), or sources > for more data other than the local authority websites (e.g. eGR, > info4local). > > Cheers > > C > > _______________________________________________ > Mailing list [email protected] > <mailto:[email protected]> > Archive, settings, or unsubscribe: > > https://secure.mysociety.org/admin/lists/mailman/listinfo/developers-public > > _______________________________________________ Mailing list [email protected] Archive, settings, or unsubscribe: https://secure.mysociety.org/admin/lists/mailman/listinfo/developers-public
