Re: [backstage] Proposal for a new API source
Sam Smith wrote: On Mon, 4 Jun 2007, Ian Forrester wrote: So how about we started to archive the RSS feeds into one large database, index them by time and date, slap a nice REST interface on them and let you guys have access to it? You mean a more complex version of: http://sebastian.foriru.co.uk/~sams/bbc/pouncing/ If anyone thinks time/date is useful, I can stick those in the feed. cheers Sam Time and date would be useful. So if I'm correct your now index all of BBC News feeds from May 13th onwards? Ian - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/
[backstage] Proposal for a new API source
Hi All, I've been thinking about what data you guys find useful and what data could be made available without getting rights clearance involved. Anyway, one thing the BBC does a lot of is RSS feeds. But there not exactly that useful for larger applications and prototypes because of lack of structure and partial nature (best way I can explain it). So how about we started to archive the RSS feeds into one large database, index them by time and date, slap a nice REST interface on them and let you guys have access to it? I imagine within a few months, you could data mine out keywords and trends. It would also make a great archive of the news at the time (suggesting we use BBC News, World Service and Sport feeds to start). Ideally in the future, you would all be able to add metadata around the original items, setup unique queries and copy the whole database. How's this sound because its something I think could be built quite easily. If people are interested I can go into the details later on the list. So what do you guys all think? Ian Forrester This e-mail is: [] private; [] ask first; [ x ] bloggable Senior Producer, BBC Backstage BC5 C3, Media Village, 201 Wood Lane, London W12 7TP e: [EMAIL PROTECTED] p: +44 (0)2080083965 - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/
Re: [backstage] Proposal for a new API source
That sounds like fun. It would be a nice research tool to pick out all the BBC News stories about a particular topic and see how it developed. Was it just the RSS feeds you wanted to stored? Because the See Also links on an news story page would be great to cross reference in the database. Is there any other story metadata floating about that could be incorporated as well? If it's easy(famous last words) then definitely build it. Sounds like a great idea to me. Daithi On 6/4/07, Ian Forrester [EMAIL PROTECTED] wrote: Hi All, I've been thinking about what data you guys find useful and what data could be made available without getting rights clearance involved. Anyway, one thing the BBC does a lot of is RSS feeds. But there not exactly that useful for larger applications and prototypes because of lack of structure and partial nature (best way I can explain it). So how about we started to archive the RSS feeds into one large database, index them by time and date, slap a nice REST interface on them and let you guys have access to it? I imagine within a few months, you could data mine out keywords and trends. It would also make a great archive of the news at the time (suggesting we use BBC News, World Service and Sport feeds to start). Ideally in the future, you would all be able to add metadata around the original items, setup unique queries and copy the whole database. How's this sound because its something I think could be built quite easily. If people are interested I can go into the details later on the list. So what do you guys all think? Ian Forrester This e-mail is: [] private; [] ask first; [ x ] bloggable Senior Producer, BBC Backstage BC5 C3, Media Village, 201 Wood Lane, London W12 7TP e: [EMAIL PROTECTED] p: +44 (0)2080083965 - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/ - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/
RE: [backstage] Proposal for a new API source
Is there any way to add the physical locations (ie, latitude and longitude) so that you can search for 'local' stories too? For example, Sports reports would have the location of the stadium, News reports would list the relevant locations etc Brian Butterworth www.ukfree.tv -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ian Forrester Sent: 04 June 2007 15:04 To: backstage@lists.bbc.co.uk Subject: [backstage] Proposal for a new API source Hi All, I've been thinking about what data you guys find useful and what data could be made available without getting rights clearance involved. Anyway, one thing the BBC does a lot of is RSS feeds. But there not exactly that useful for larger applications and prototypes because of lack of structure and partial nature (best way I can explain it). So how about we started to archive the RSS feeds into one large database, index them by time and date, slap a nice REST interface on them and let you guys have access to it? I imagine within a few months, you could data mine out keywords and trends. It would also make a great archive of the news at the time (suggesting we use BBC News, World Service and Sport feeds to start). Ideally in the future, you would all be able to add metadata around the original items, setup unique queries and copy the whole database. How's this sound because its something I think could be built quite easily. If people are interested I can go into the details later on the list. So what do you guys all think? Ian Forrester This e-mail is: [] private; [] ask first; [ x ] bloggable Senior Producer, BBC Backstage BC5 C3, Media Village, 201 Wood Lane, London W12 7TP e: [EMAIL PROTECTED] p: +44 (0)2080083965 - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/ No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.472 / Virus Database: 269.8.7/830 - Release Date: 03/06/2007 12:47 No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.472 / Virus Database: 269.8.7/830 - Release Date: 03/06/2007 12:47 - Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html. Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/
Re: [backstage] Proposal for a new API source
Ian Forrester wrote, reordered slightly:
So what do you guys all think?
Sounds like a great idea; especially as I've been doing this for the BBC
News front page (similar to the RSS feed, I guess) since 2005? :-)
I imagine within a few months, you could data mine out keywords and
trends.
You certainly could. :) Perhaps with a tag cloud:
http://www.dracos.co.uk/work/bbc-news-archive/tardis/tag-cloud/
Or watching how the word Iran is used over time:
http://www.dracos.co.uk/work/bbc-news-archive/tardis/search-headline/?s=iran
Or which pictures the BBC decides to illustrate stories about the Olympics:
http://www.dracos.co.uk/work/bbc-news-archive/tardis/image-search/?s=olympics
It would also make a great archive of the news at the time
Here's my archive of the front page of BBC News for the past 6 months or so:
http://www.dracos.co.uk/work/bbc-news-archive/archive.php
Ideally in the future, you would all be able to add metadata around the
original items, setup unique queries and copy the whole database.
Sadly, not sure I have the bandwidth for that. ;) You can create a unique
timeline of how an event appeared on the front page:
http://www.dracos.co.uk/work/bbc-news-archive/tardis/timeline/?t=Richard+Hammond%27s+crash
So how about we started to archive the RSS feeds into one large database,
index them by time and date, slap a nice REST interface on them and let
you guys have access to it?
Seems odd for you to have to export the data from somewhere it should
already be a database to RSS to import it back in again - perhaps public
read access to the actual BBC News content management database would be a
bit hard to justify. ;-) Apologies that my dataset doesn't have an API
beyond what's there; I'd be happy to add things if people asked (no-one ever
has :) ) and I had the time. But I can do database queries, e.g. what were
the BBC News front page main headlines on New Years Day 2006?
--8---
msbbc= select url,headline,start,description from headline where
date_trunc('day', start)='2006-01-01' and type='headline' order by start;
/1/hi/uk/4570980.stm | Revellers ready to welcome 2006 |
2006-01-01 00:00:00 | Millions in the UK prepare to see in the New Year,
while in London a Tube strike appears not to have caused chaos.
/1/hi/uk/4570980.stm | Revellers welcome 2006 across UK |
2006-01-01 00:05:00 | Millions in the UK celebrate the New Year, while in
London a Tube strike appears not to have caused chaos.
/1/hi/uk/4570980.stm | Revellers welcome 2006 across UK |
2006-01-01 04:35:00 | Millions in the UK celebrate the New Year, with
parties and spectacular firework displays in many cities.
/1/hi/health/4529762.stm | Rift over greater hospital choice |
2006-01-01 13:40:00 | People in England are given a choice of hospitals for
non-emergency surgery, but unions voice concerns.
/1/hi/world/europe/4573572.stm | Ukraine gas crisis 'could hit EU' |
2006-01-01 16:20:00 | Gas exports to the European Union could be hit by the
axeing of Russian supplies to Ukraine, officials say.
/1/hi/world/europe/4573572.stm | Ukraine gas row hits EU supplies |
2006-01-01 17:40:00 | EU countries start to suffer knock-on cuts in gas
imports as Russia axes supplies to Ukraine.
/1/hi/world/europe/4573572.stm | Ukraine gas row hits EU supplies |
2006-01-01 17:45:00 | EU countries start to suffer knock-on cuts in gas
imports as Russia axes supplies to Ukraine in a row over prices.
--8---
Might be useful to someone, you never know!
--
ATB,
Matthew | http://www.dracos.co.uk/
-
Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please
visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html.
Unofficial list archive: http://www.mail-archive.com/backstage@lists.bbc.co.uk/
