Re: peculiar to BBC Radio Wales?
artisticforge . artisticfo...@gmail.com wrote: concerning the below: INFO: 1 Matching Programmes INFO: Programme not in history WARNING: Page parsing may fail with HTML::Parser versions before 3.71. You have version 3.69. I am running Debian Wheezy and Debian has not upgrade those modules as of yet. I am an Electrical Engineer, B.S.E.E., University of Illinois, Urbana-Champaign, IL. so I do know how to read and I am aware of possible issues that the fore mentioned mismatch may cause. I had all ready proven to myself that the mismatch was not the issue. Then you should have said so. No-one here is clairvoyant. -- Jeremy Nicoll - my opinions are my own. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
Hello Like get_iplayer, I to may real soon now be obsolete as in dead. So forgive me for not filling in a the required blanks when posting. I have survived three events any one of which should have killed me. Trust me, there things worse than death. I spend my days either inspecting the ceiling or seating in a wheelchair. Morphine, keeps the pain at bay. Sent from my iPad terry l. ridder On Nov 17, 2014, at 4:54, Jeremy Nicoll - ml get_iplayer jn.ml.gti...@wingsandbeaks.org.uk wrote: artisticforge . artisticfo...@gmail.com wrote: concerning the below: INFO: 1 Matching Programmes INFO: Programme not in history WARNING: Page parsing may fail with HTML::Parser versions before 3.71. You have version 3.69. I am running Debian Wheezy and Debian has not upgrade those modules as of yet. I am an Electrical Engineer, B.S.E.E., University of Illinois, Urbana-Champaign, IL. so I do know how to read and I am aware of possible issues that the fore mentioned mismatch may cause. I had all ready proven to myself that the mismatch was not the issue. Then you should have said so. No-one here is clairvoyant. -- Jeremy Nicoll - my opinions are my own. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
Hello
The patch suggested works.
The work now is to incorporate this into get_iplayer so that a ms4
failure falls over to try ms5.
The patched get_iplayer has downloaded Roy Noble and Beverley's World
of Music from BBC radio Wales.
On Mon, Nov 17, 2014 at 1:22 AM, Vangelis forthnet
northmed...@the.forthnet.gr wrote:
On Mon Nov 17 00:31:24 GMT 2014, artisticforge wrote:
today, of the three BBC Radio Wales programmes that I normally get
all are having to be downloaded as liveradio with radiomode=wma
the three programmes are Roy Noble,
(snip)
I have a gut feeling that this is a permanent change.
Greetings, Terry :-)
Unfortunately, your gut feeling is right;
this is another prelude of more changes to come;
this issue of yours has the same root cause as the
one reported in the following support forum thread:
https://squarepenguin.co.uk/forums/topic/downloads-from-bbc-radio-nan-gaidheal-and-radio-ulster-have-failed/
The BBC are testing (?) the new changes starting with the Nations Radios...
You being this highly educated, I'll provide some technical details that I'm
sure
will puzzle the non-savvy...
Taking as an example the Roy Noble radio programme, with PID=b04nj385,
the legacy XML playlist that GiP first tries:
http://www.bbc.co.uk/iplayer/playlist/b04nj385
returns: noItems reason=preAvailability/.
GiP has been recently patched and so then tries
the new JSON playlist:
http://www.bbc.co.uk/programmes/b04nj385/playlist.json
This, when parsed, returns: vPID=b04nj37m.
vPID string is used to rertrieve the stream details.
GiP up to now uses exclusively the legacy
mediaselector/4 API (I'll call it ms4 for brevity):
http://www.bbc.co.uk/mediaselector/4/mtis/stream/b04nj37m
The BBC have left but the wma stream data (48kbps bitrate),
the rtmp stream data previously also present there have vanished...
So GiP only sees the wma radiomode as being available for
recording...
The Beeb have in turn implemented the new mediaselector/5
(ms5) API for this radio show and, evidently, many others from
the Nations radios.
This new ms5 URL varies according to the client device the stream
is destined for - this is the mediaset part of the URL; for desktops
laptops, mediaset=pc and the URL looks like:
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/b04nj37m
As you can see, ONLY THE RTMP STREAMS are therein
(radiomodes flashaacstd flashaaclow). This is because, as they have
already announced, the wma streams will be axed in the end of the year...
For info, other mediasets that I have found browsing their code are:
mediaset/journalism-pc/
mediaset/mobile-phone-basic/
mediaset/mobile-phone-main/
mediaset/mobile-tablet-main/
mediaset/journalism-http-tablet/
and others...
So, for a mobile tablet the ms5 URL would be:
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/mobile-tablet-main/vpid/b04nj37m
(you'll notice therein the details of the akamai_hls_open streams,
delivered over http - they have also heralded the implementation of
AdobeHDS streams for AOD sometime in 2015).
Now, if you still want the rtmp stream of Roy Noble dumped to disk,
you can:
1. Manually retrieve the stream parameters from the ms5 page and
then manually compose a working rtmpdump command - I have posted
templates for both ak ll CDNs some weeks ago in the list:
http://lists.infradead.org/pipermail/get_iplayer/attachments/20141007/8253d96d/attachment.txt
2. patch your local copy of GiP to use the ms5 URL -
Make a copy of your version of the script and edit it with an editor.
Sadly I am not a coder, so I had to apply a butcher method myself
(this is because from the start of September, the majority of visual content
(usually clips of TV programmes) offered to non-UK users of iplayer
is using exclusively the ms5 infrastructure).
In the get_stream_data subroutine, change
my $media_stream_data_prefix =
'http://www.bbc.co.uk/mediaselector/4/mtis/stream/'; # $verpid
to
my $media_stream_data_prefix =
'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/';
# $verpid
Also, the ms5 pages do not contain the kind= entry, but the supplier=
entry
with equal value can be used in its place.
So, some
$cattribs-{kind}
entries in the GiP code (inside the get_stream_data_cdn subroutine)
for flash on demand streams must be changed to
$cattribs-{supplier}
BUT FIRST, comment out the following code:
# Get authstring from more specific mediaselector if this mode is
specified - fails sometimes otherwise
# if ( $cattribs-{authString} $cattribs-{supplier} =~
/^(limelight|akamai|level3|sis|iplayertok)$/ (grep /^$mode$/, (split /,/,
$mattribs-{modelist})) ) {
# Build URL
# my $media_stream_data_prefix =
'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/';
# my $url =
Re: peculiar to BBC Radio Wales?
On 17/11/2014 00:31, artisticforge . wrote: today, of the three BBC Radio Wales programmes that I normally get all are having to be downloaded as liveradio with radiomode=wma the three programmes are Roy Noble, Celtic Heartbeat and Beverley's World of Music First cut at a fix now in Git: https://github.com/dinkypumpkin/get_iplayer/wiki/installation#githead There have a been several similar problems reported lately (all with nations radio programmes so far), enough to be more than coincidental. I'm assuming that the mediaselector/4 API is on the way to being killed, or left to rot, at least in part. Whatever the case, some RTMP streams are disappearing. So, for anyone using this patch: - I've changed get_iplayer to use the mediaselector/5 API by default. If you don't know what mediaselector/5 API means, all you need to know is that it is a newer BBC interface for retrieving media stream parameters. It should return RTMP stream data that is missing from the older mediaselector/4 API for some programmes. - AFAICT, this change seems to work for iPlayer content and BBC news site videos. In the past, there were some other esoteric audio/video things that get_iplayer could download. I don't remember what they were specifically, or if they are still around. If you find some non-iPlayer content along those lines that no longer works, feel free to post it. However, my attitude is that anything that is not iPlayer content can be thrown under the bus it it's any trouble to fix. - wma mode (radio) and flashlow mode (tv) are not available. WMA is going bye-bye in 6 weeks, so you should be weaning yourself off it anyway. Lower-quality modes for video are now only available in the mobile device mediasets. If you don't know what mediaset means in this context, all you need to is that it is a construct used to segment the available media streams into groups for different types of devices, e.g., mobile vs desktop. It may be possible to bring in the low-quality video modes from mobile mediasets, but that is a headache for another time. - Use --mediaselector=4 to return to the old behaviour, e.g., in order to get wma or flashlow mode. There is no guarantee it will work, however. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
Hello; The new patch appears to be working just fine. I fired up get-iplayer-automater under snow leopard on my one mac-mini. i replaced the get_iplayer with the new patched version. i see no reason why it will not work, but going to test more just to make sure. get_iplayer-patch-ms5 is running excellent under Debian Linux Wheezy. On Mon, Nov 17, 2014 at 2:01 PM, dinkypumpkin dinkypump...@gmail.com wrote: On 17/11/2014 00:31, artisticforge . wrote: today, of the three BBC Radio Wales programmes that I normally get all are having to be downloaded as liveradio with radiomode=wma the three programmes are Roy Noble, Celtic Heartbeat and Beverley's World of Music First cut at a fix now in Git: https://github.com/dinkypumpkin/get_iplayer/wiki/installation#githead There have a been several similar problems reported lately (all with nations radio programmes so far), enough to be more than coincidental. I'm assuming that the mediaselector/4 API is on the way to being killed, or left to rot, at least in part. Whatever the case, some RTMP streams are disappearing. So, for anyone using this patch: - I've changed get_iplayer to use the mediaselector/5 API by default. If you don't know what mediaselector/5 API means, all you need to know is that it is a newer BBC interface for retrieving media stream parameters. It should return RTMP stream data that is missing from the older mediaselector/4 API for some programmes. - AFAICT, this change seems to work for iPlayer content and BBC news site videos. In the past, there were some other esoteric audio/video things that get_iplayer could download. I don't remember what they were specifically, or if they are still around. If you find some non-iPlayer content along those lines that no longer works, feel free to post it. However, my attitude is that anything that is not iPlayer content can be thrown under the bus it it's any trouble to fix. - wma mode (radio) and flashlow mode (tv) are not available. WMA is going bye-bye in 6 weeks, so you should be weaning yourself off it anyway. Lower-quality modes for video are now only available in the mobile device mediasets. If you don't know what mediaset means in this context, all you need to is that it is a construct used to segment the available media streams into groups for different types of devices, e.g., mobile vs desktop. It may be possible to bring in the low-quality video modes from mobile mediasets, but that is a headache for another time. - Use --mediaselector=4 to return to the old behaviour, e.g., in order to get wma or flashlow mode. There is no guarantee it will work, however. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer -- terry l. ridder ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
artisticforge . artisticfo...@gmail.com wrote: INFO: 1 Matching Programmes INFO: Programme not in history WARNING: Page parsing may fail with HTML::Parser versions before 3.71. You have version 3.69. I wondered if this was relevant, as it suggests you're either not using a recent enough version of perl installed by g_ip's installer (which would also install uptodate versions of perl 'modules'), or you've manually installed perl (or had it already) and not kept the modules uptodate. Or if you're not using get_iplayer 2.90 - you should really tell us... But I do have 2.90 and an uptodate perl etc, so I tried a fetch including --pid b04nj385 --mode crap --verbose The crap is there deliberately, because I know that even if everything else is ok there won't be a crap mode available, so the fetch won't go ahead - in other words it's an easy way to see verbose command output without going ahead with the actual fetch. Anyway I also only saw a single mode - wma1 - being offered, so the obvious INFO message isn't relevant. -- Jeremy Nicoll - my opinions are my own. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
hello concerning the below: INFO: 1 Matching Programmes INFO: Programme not in history WARNING: Page parsing may fail with HTML::Parser versions before 3.71. You have version 3.69. I am running Debian Wheezy and Debian has not upgrade those modules as of yet. I am an Electrical Engineer, B.S.E.E., University of Illinois, Urbana-Champaign, IL. so I do know how to read and I am aware of possible issues that the fore mentioned mismatch may cause. I had all ready proven to myself that the mismatch was not the issue. Concerning get_iplayer, I am using get_iplayer-bb82110. which is get_iplayer-2.90 with further editing. Retrieved from https://github.com/dinkypumpkin/get_iplayer The bottomline to your posting is that things are changing at BBC Radio Wales and perhaps others. I have a gut feeling that really soon get_iplayer will be rendered obsolete. On Sun, Nov 16, 2014 at 7:55 PM, Jeremy Nicoll - ml get_iplayer jn.ml.gti...@wingsandbeaks.org.uk wrote: artisticforge . artisticfo...@gmail.com wrote: INFO: 1 Matching Programmes INFO: Programme not in history WARNING: Page parsing may fail with HTML::Parser versions before 3.71. You have version 3.69. I wondered if this was relevant, as it suggests you're either not using a recent enough version of perl installed by g_ip's installer (which would also install uptodate versions of perl 'modules'), or you've manually installed perl (or had it already) and not kept the modules uptodate. Or if you're not using get_iplayer 2.90 - you should really tell us... But I do have 2.90 and an uptodate perl etc, so I tried a fetch including --pid b04nj385 --mode crap --verbose The crap is there deliberately, because I know that even if everything else is ok there won't be a crap mode available, so the fetch won't go ahead - in other words it's an easy way to see verbose command output without going ahead with the actual fetch. Anyway I also only saw a single mode - wma1 - being offered, so the obvious INFO message isn't relevant. -- Jeremy Nicoll - my opinions are my own. -- terry l. ridder ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: peculiar to BBC Radio Wales?
On Mon Nov 17 00:31:24 GMT 2014, artisticforge wrote:
today, of the three BBC Radio Wales programmes that I normally get
all are having to be downloaded as liveradio with radiomode=wma
the three programmes are Roy Noble,
(snip)
I have a gut feeling that this is a permanent change.
Greetings, Terry :-)
Unfortunately, your gut feeling is right;
this is another prelude of more changes to come;
this issue of yours has the same root cause as the
one reported in the following support forum thread:
https://squarepenguin.co.uk/forums/topic/downloads-from-bbc-radio-nan-gaidheal-and-radio-ulster-have-failed/
The BBC are testing (?) the new changes starting with the Nations Radios...
You being this highly educated, I'll provide some technical details that I'm
sure
will puzzle the non-savvy...
Taking as an example the Roy Noble radio programme, with PID=b04nj385,
the legacy XML playlist that GiP first tries:
http://www.bbc.co.uk/iplayer/playlist/b04nj385
returns: noItems reason=preAvailability/.
GiP has been recently patched and so then tries
the new JSON playlist:
http://www.bbc.co.uk/programmes/b04nj385/playlist.json
This, when parsed, returns: vPID=b04nj37m.
vPID string is used to rertrieve the stream details.
GiP up to now uses exclusively the legacy
mediaselector/4 API (I'll call it ms4 for brevity):
http://www.bbc.co.uk/mediaselector/4/mtis/stream/b04nj37m
The BBC have left but the wma stream data (48kbps bitrate),
the rtmp stream data previously also present there have vanished...
So GiP only sees the wma radiomode as being available for
recording...
The Beeb have in turn implemented the new mediaselector/5
(ms5) API for this radio show and, evidently, many others from
the Nations radios.
This new ms5 URL varies according to the client device the stream
is destined for - this is the mediaset part of the URL; for desktops
laptops, mediaset=pc and the URL looks like:
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/b04nj37m
As you can see, ONLY THE RTMP STREAMS are therein
(radiomodes flashaacstd flashaaclow). This is because, as they have
already announced, the wma streams will be axed in the end of the year...
For info, other mediasets that I have found browsing their code are:
mediaset/journalism-pc/
mediaset/mobile-phone-basic/
mediaset/mobile-phone-main/
mediaset/mobile-tablet-main/
mediaset/journalism-http-tablet/
and others...
So, for a mobile tablet the ms5 URL would be:
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/mobile-tablet-main/vpid/b04nj37m
(you'll notice therein the details of the akamai_hls_open streams,
delivered over http - they have also heralded the implementation of
AdobeHDS streams for AOD sometime in 2015).
Now, if you still want the rtmp stream of Roy Noble dumped to disk,
you can:
1. Manually retrieve the stream parameters from the ms5 page and
then manually compose a working rtmpdump command - I have posted
templates for both ak ll CDNs some weeks ago in the list:
http://lists.infradead.org/pipermail/get_iplayer/attachments/20141007/8253d96d/attachment.txt
2. patch your local copy of GiP to use the ms5 URL -
Make a copy of your version of the script and edit it with an editor.
Sadly I am not a coder, so I had to apply a butcher method myself
(this is because from the start of September, the majority of visual content
(usually clips of TV programmes) offered to non-UK users of iplayer
is using exclusively the ms5 infrastructure).
In the get_stream_data subroutine, change
my $media_stream_data_prefix =
'http://www.bbc.co.uk/mediaselector/4/mtis/stream/'; # $verpid
to
my $media_stream_data_prefix =
'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/';
# $verpid
Also, the ms5 pages do not contain the kind= entry, but the supplier=
entry
with equal value can be used in its place.
So, some
$cattribs-{kind}
entries in the GiP code (inside the get_stream_data_cdn subroutine)
for flash on demand streams must be changed to
$cattribs-{supplier}
BUT FIRST, comment out the following code:
# Get authstring from more specific mediaselector if this mode is
specified - fails sometimes otherwise
# if ( $cattribs-{authString} $cattribs-{supplier} =~
/^(limelight|akamai|level3|sis|iplayertok)$/ (grep /^$mode$/, (split /,/,
$mattribs-{modelist})) ) {
# Build URL
# my $media_stream_data_prefix =
'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/';
# my $url =
$media_stream_data_prefix.$mattribs-{verpid}/$mattribs-{service}/$cattribs-{supplier}?cb=.(
sprintf %05.0f, 9*rand(0) );
# my $xml = main::request_url_retry( main::create_ua( 'desktop' ), $url,
3, undef, undef, 1 );
# main::logger \n$xml\n if $opt-{debug};
# get new set of connection attributes from the new xml data
# my $new_mattribs = (parse_metadata( $xml ))[0];
# my $new_cattribs = $new_mattribs-{connections}[0];
# Override elements from more specific
