Re: get_iplayer repair update #1
Thank you for your hard work on get_iplayer. Alan. On 1 November 2014 00:45, dinkypumpkin dinkypump...@gmail.com wrote: get_iplayer has been more or less repaired, but there are still some wounds. I'm going to release what I have on Sunday. I'm on the road next week, so I've run out of time to do more for the time being. Consider it a stopgap until progress can be made on other fronts. This is where things are: 1. I've disabled code related to the discontinued feeds, so you shouldn't get any more bogus values in your metadata tags. You should also see thumbnails again in files 7 days old downloaded via PID. 2. The new release will support entry of multiple PIDs. 3. I've more or less restored the 7 day cache for TV and radio. There are still some holes in it: a. It is not possible to search for audiodescribed versions of programmes. I haven't been able to source that information. If anyone has any clues on the subject, chime in - but not if your suggestion is to scrape the iPlayer site. That isn't on the table right just yet. You can still download audiodescribed versions, but you'll have to look for them on the iPlayer site. Signed versions should still be flagged in the get_iplayer cache, but some may be missing. Again, check the iPlayer site if in doubt. I've changed get_iplayer to always scrape the related episode page to look for audiodescribed/signed versions when requested, so hopefully more downloads will be successful. I found a number of cases where the playlist data for recent programmes didn't contain identifiers for audiodescribed versions even though they existed on the iPlayer site. b. It is not possible to search radio programmes by category. TV programmes still have category information. There is a source for radio category information, but it uniformly foundered on Radio 4 and Radio 4 Extra, which is where the categories are most meaningful. I know that is going to break some PVR searches, but the alternative is a support headache I can't absorb. c. I can't vouch that every programme from the previous 7 days will show up in the cache. As always, you can use the PID for any programme not in the cache. By the same token, I can't vouch that every programme in the cache will be downloadable. The new feeds contain noticeably more programmes, some due to the inclusion of web-only stuff. With the heavier load, cache refreshes are noticeably slower than with the old feeds, ca. 90 seconds for me for tv+radio. 2. The more-or-less restored cache depends on some old data feeds lingering at the BBC. Recent events have taught us that they could disappear without warning, so I've implemented a fallback mechanism. There will be a new option that will switch the cache to refresh from the channel schedule pages instead of the old data feeds. However, this fallback is also limited: a. It is not possible to search for audiodescribed or signed versions of programmes. That information isn't in the schedule pages. b. It is not possible to search TV or radio programmes by category. Again, that information isn't in the schedule pages. c. Cache refresh is slow, ca. 4+ minutes for a full TV and radio refresh for me. The time could be cut by about 1/3 by removing regional TV channel variations, but it cuts out 50+ programmes, so I've left them in for the present. d. It appears that fewer programmes from the previous 7 days get cached compared to the feeds. Part of that is because the schedule pages don't show most web-only programmes. Part of it may also be because I'm checking availability info in the schedule pages more strictly than whatever produces the data feeds. Again, you can use the PID for anything not in the cache. e. The only plus to using the schedule pages to populate the cache is that it becomes possible to expand your cache out to 30 days. It seems to work OK, if you have 10-15 minutes to refresh your cache. There will be an option for this. f. I've given you enough rope to hang yourself, but don't put this fallback option into regular use unless it becomes necessary - seriously. It's only there to avoid weeks like this one. I won't be interested in hearing how slow it is or how it doesn't locate some particular programme. And for pete's sake *don't* use it with the Web PVR. If you insist on playing around with it, you'll probably want to bump up --expiry to some gigantic number and refresh your cache manually as needed. 3. Looking further ahead Some things that have been floated here in the past few days: a. Programme data services: If somebody implements something along these lines, I'm sure get_iplayer could be integrated with it. It's clear that get_iplayer would never be able to access Nitro if and when it's ever opened up. But, if somebody can repackage Nitro data for wider use, that would be pretty useful. b. iPlayer site scraping: This could also be the foundation of a
Re: get_iplayer repair update #1
On 01/11/2014 00:45, dinkypumpkin wrote: get_iplayer has been more or less repaired, but there are still some wounds... A huge THANK YOU for your previous, and continuing, efforts to keep get_iPlayer alive. It is much appreciated by many people. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: get_iplayer repair update #1
Can I also say a massive thank you to Dinky and others who have worked in double quick time to get a fix. I have used get_iplayer for several years but only joined the mailing list when disaster seemed to strike late last week. The response has been a fantastic demonstration of open source community work, and I hope a more permanent solution will be possible in the coming weeks (or months). Thank you also to Jon for maintaining the ppa for ubuntu users (like me -- I also use get_iplayer on the raspberry pi). Reading the posts has made me want to start learning perl or python, but I am probably too old (50) and am stuck at present learning scratch with my younger son (aged 6). On Sat, 1 Nov, 2014 at 9:52 PM, Peter S Kirk peter.k...@isauk.biz wrote: On 1 Nov 2014 at 1:52, Jeremy Nicoll - ml get_iplayer Jeremy Nicoll - ml get_iplayer jn.ml.gti...@wingsandbeaks.org.uk wrote: dinkypumpkin dinkypump...@gmail.com wrote: get_iplayer has been more or less repaired, but there are still some wounds. I'm going to release what I have on Sunday This is excellent news, and I have to say I'm impressed by the amount you've managed to do in such a short time. (I realise that we may have been lucky with the time of year; I should think that it's a bad time for any sort of pumpkin to be out of the ... house? shed? field? ... wherever a pumpkin normally feels at home. I've seen some dreadful sights in the last 12 hours - pumpkins being carried around by witches etc...) Dinky, As Jeremy says, a huge thank you for the effort you and others are putting into work arounds for the BBC changes. Very much appreciated. I too am pleased you survived another Halloween uncut :) Jeremy, well done - very amusing and made me laugh :) Cheers, Peter PS Ignore previous copy.com referal link for bonus - it no longer works as copy have implemented a max five referals policy :( I have other referal links which will give the 5GB bonus - PM me. Sorry if you have joined and not received bonus 5GB, with details and I will make a complaint as I was not informed of this change. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
get_iplayer repair update #1
get_iplayer has been more or less repaired, but there are still some wounds. I'm going to release what I have on Sunday. I'm on the road next week, so I've run out of time to do more for the time being. Consider it a stopgap until progress can be made on other fronts. This is where things are: 1. I've disabled code related to the discontinued feeds, so you shouldn't get any more bogus values in your metadata tags. You should also see thumbnails again in files 7 days old downloaded via PID. 2. The new release will support entry of multiple PIDs. 3. I've more or less restored the 7 day cache for TV and radio. There are still some holes in it: a. It is not possible to search for audiodescribed versions of programmes. I haven't been able to source that information. If anyone has any clues on the subject, chime in - but not if your suggestion is to scrape the iPlayer site. That isn't on the table right just yet. You can still download audiodescribed versions, but you'll have to look for them on the iPlayer site. Signed versions should still be flagged in the get_iplayer cache, but some may be missing. Again, check the iPlayer site if in doubt. I've changed get_iplayer to always scrape the related episode page to look for audiodescribed/signed versions when requested, so hopefully more downloads will be successful. I found a number of cases where the playlist data for recent programmes didn't contain identifiers for audiodescribed versions even though they existed on the iPlayer site. b. It is not possible to search radio programmes by category. TV programmes still have category information. There is a source for radio category information, but it uniformly foundered on Radio 4 and Radio 4 Extra, which is where the categories are most meaningful. I know that is going to break some PVR searches, but the alternative is a support headache I can't absorb. c. I can't vouch that every programme from the previous 7 days will show up in the cache. As always, you can use the PID for any programme not in the cache. By the same token, I can't vouch that every programme in the cache will be downloadable. The new feeds contain noticeably more programmes, some due to the inclusion of web-only stuff. With the heavier load, cache refreshes are noticeably slower than with the old feeds, ca. 90 seconds for me for tv+radio. 2. The more-or-less restored cache depends on some old data feeds lingering at the BBC. Recent events have taught us that they could disappear without warning, so I've implemented a fallback mechanism. There will be a new option that will switch the cache to refresh from the channel schedule pages instead of the old data feeds. However, this fallback is also limited: a. It is not possible to search for audiodescribed or signed versions of programmes. That information isn't in the schedule pages. b. It is not possible to search TV or radio programmes by category. Again, that information isn't in the schedule pages. c. Cache refresh is slow, ca. 4+ minutes for a full TV and radio refresh for me. The time could be cut by about 1/3 by removing regional TV channel variations, but it cuts out 50+ programmes, so I've left them in for the present. d. It appears that fewer programmes from the previous 7 days get cached compared to the feeds. Part of that is because the schedule pages don't show most web-only programmes. Part of it may also be because I'm checking availability info in the schedule pages more strictly than whatever produces the data feeds. Again, you can use the PID for anything not in the cache. e. The only plus to using the schedule pages to populate the cache is that it becomes possible to expand your cache out to 30 days. It seems to work OK, if you have 10-15 minutes to refresh your cache. There will be an option for this. f. I've given you enough rope to hang yourself, but don't put this fallback option into regular use unless it becomes necessary - seriously. It's only there to avoid weeks like this one. I won't be interested in hearing how slow it is or how it doesn't locate some particular programme. And for pete's sake *don't* use it with the Web PVR. If you insist on playing around with it, you'll probably want to bump up --expiry to some gigantic number and refresh your cache manually as needed. 3. Looking further ahead Some things that have been floated here in the past few days: a. Programme data services: If somebody implements something along these lines, I'm sure get_iplayer could be integrated with it. It's clear that get_iplayer would never be able to access Nitro if and when it's ever opened up. But, if somebody can repackage Nitro data for wider use, that would be pretty useful. b. iPlayer site scraping: This could also be the foundation of a programme data service instead of Nitro. It is also the only real hope for get_iplayer to regain a full-featured desktop
RE: get_iplayer repair update #1
Wireshark MAY tell you that they're using the Nitro API with a defined key I couldn't possibly comment though. -Original Message- From: get_iplayer [mailto:get_iplayer-boun...@lists.infradead.org] On Behalf Of Steven Maude Sent: 31 October 2014 18:08 To: get_iplayer Subject: Re: get_iplayer repair update #1 On 01/11/2014 00:45, dinkypumpkin wrote: Some things that have been floated here in the past few days: snip lots Yet another one, which I'll mention again here as it was lost in the deluge. Does anyone know how the BBC's mobile apps work? Are they simply web-based frontends or do they use some private API? If the latter, then it might be possible to use that to retrieve metadata, with the advantages that it's unlikely to have support dropped overnight or be as fragile as scraping. This post gives some guidance on how to investigate: http://timrogers.uk/2014/07/12/discovering-private-apis-with-charles-app/ Anyway, great work; look forward to the update and it's definitely appreciated. Hope you get chance to relax a bit over the weekend! ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: get_iplayer repair update #1
dinkypumpkin dinkypump...@gmail.com wrote: get_iplayer has been more or less repaired, but there are still some wounds. I'm going to release what I have on Sunday This is excellent news, and I have to say I'm impressed by the amount you've managed to do in such a short time. (I realise that we may have been lucky with the time of year; I should think that it's a bad time for any sort of pumpkin to be out of the ... house? shed? field? ... wherever a pumpkin normally feels at home. I've seen some dreadful sights in the last 12 hours - pumpkins being carried around by witches etc...) c. External search ... I'll have my pony now, thanks. Whoosh? That's certainly over my head. Unless it's an insight into a pumpkin's dietary preferences? Regarding cacheing from feeds vv scraping... cacheing seems to me to have been good for capturing a snapshot of all the available programmes at a point in time. But for anyone who's a regular user of get_iplayer there's a lot of overlap in the cache's contents from one version/refresh to the next. I'm not so sure that scraping the schedules to see what the BBC plan to transmit in just the following day, say, would be so bad, especially if it was limited by user selection to just the channels/stations that a user was most interested in. It could mean that it'd take a few days for such a user to build up a list of programmes that should be available, but then actual availability checks per programme could wait until someone tried to download one. -- Jeremy Nicoll - my opinions are my own. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
