Re: OT: Playing GiP data wirelessly.

[Dave Lambley]

> On 05/09/2021 09:24 Budge  wrote:

> Only two speakers installed flush in ceiling at transition with conical roof 
> and all others removed for change of use.  I agree that there is no point in 
> trying to create a "stereo" sound image and so am just trying to think 
> through what would be the simplest mono configuration.  I now only have 
> stereo amplifiers but several available and in various powers.

A wirelessly connected stereo pair is possible (at least according to a 
marketing director), 
https://www.theregister.com/2014/01/14/imagination_caskeid_wifi_audio_synchronization/

I don't know if Imgtec sold these chips to anyone, but I notice Denon are 
selling speakers which claim to do this.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: "ERROR: Failed to download URL "

[Dave Lambley]

> On 03/13/2021 1:50 PM Mark Carroll  wrote:
> 
>  
> On 13 Mar 2021, mable syrup wrote:
> 
> > Started getting these recently for TV programmes.  Anyone know:
> > (1) Whether there's a fix in the offing?
> > (2) Do I need to report as an issue on Github?
> > (3) Is there a workaround?
> 
> I have not noticed any indication that anything other than useful bugfix
> PRs is presently welcomed by the maintainer. Reports of issues tend to
> be summarily closed. We hope and wait for fixes and, at least for major
> issues, generally they come promptly, which is great. If you know some
> Perl, the source code is quite amenable to local adjustments.

There is no-one maintaining get-iplayer at the moment. GitHub has been 
configured to automatically close issues with no human intervention. 
dinkypumpkin does say that if you have a bug fix, he will merge it, but new 
features won't be merged unless you wish to take over as maintainer.

See 
https://github.com/get-iplayer/get_iplayer/commit/34f400ac391bdc1880362420b266d349b86621c7
 and https://github.com/get-iplayer/get_iplayer/wiki/contribute#code

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: OT Question on audio downloads from youtube

[Dave Lambley]

> On 09/24/2020 10:25 AM Budge  wrote:
> 
>  
> On 23/09/2020 09:55, Roger Bell_West wrote:
> > On Wed, Sep 23, 2020 at 12:16:10AM +0100, budge wrote:
> >> I have now found that minimserver, which runs in the NAS and serves the 
> >> music
> >> data to my LAN and thence to my renderers, is not supported by .opus.  I 
> >> can
> >> use AAC files so will download them to .aac or  should that be .m4a.  No
> >> hardship and glad I found out now.
> > 
> > It can be both. aac is an encoding format; m4a is a container format.
> > For example if I ffprobe the latest Brain of Britain (obtained via
> > GIP) I get:
> > 
> > Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz,
> > stereo, fltp, 320 kb/s (default)
> > 
> > This is an AAC-encoded stream in an m4a container.
> > 
> 
> Hi Roger,
> Life in never simple!  My choice of .opus was frustrated by lack of
> support for my media server and author advises he will not likely be
> adding .opus.
> 
> I therefore went with .aac as the next best option.
> 
> I now find that if I wish to include thumbnails I can only use .m4a (or
> .mp4).
> 
> I appreciate thumbnails do nothing for sound but they do make searching
> easier so looks like .m4a is the way to go.  Pity I just downloaded 18
> files to .aac.
> 
> My only other comment for now is that the metadata that comes with the
> download is very poor and very short on the info I have been used to.  I
> think the problem is that whilst CD tagging is fairly well supported,
> the YouTube videos are often prepared for a different audience but I had
> hoped that at least the artists would be named.
> If there are ways of improving this I would appreciate advice.

If you have ffmpeg and AtomicParsley installed, youtube-dl can repack the audio 
in to an .m4a for you, and give you some very rudimentary tagging.

Try,

./youtube-dl -x --embed-thumbnail --add-metadata -f m4a 
https://www.youtube.com/watch?v=1L993HNAa8M

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: OT Question on audio downloads from youtube

[Dave Lambley]

> On 09/23/2020 10:55 AM Roger Bell_West  wrote:
> 
>  
> On Wed, Sep 23, 2020 at 12:16:10AM +0100, budge wrote:
> >I have now found that minimserver, which runs in the NAS and serves the music
> >data to my LAN and thence to my renderers, is not supported by .opus.  I can
> >use AAC files so will download them to .aac or  should that be .m4a.  No
> >hardship and glad I found out now.
> 
> It can be both. aac is an encoding format; m4a is a container format.
> For example if I ffprobe the latest Brain of Britain (obtained via
> GIP) I get:
> 
> Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz,
> stereo, fltp, 320 kb/s (default)
> 
> This is an AAC-encoded stream in an m4a container.

If you have ffmpeg installed, "-x" should cause the AAC to be repackaged into 
an m4a file.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: OT Question on audio downloads from youtube

[Dave Lambley]

Hi Budge,

> On 09/21/2020 11:56 PM budge  wrote:
> 
>  
> Please forgive the OT question but I am seeking advice on sensible 
> download format for saving the audio from youtube videos which are 
> available to supplement specific items from my early music collection.
> 
> My preferred solution is to download the audio using youtube-dl and I 
> know I could then encode the file as a flac file but this creates huge 
> files, albeit lossless, so my question is, is there any point in doing 
> this if the initial audio quality is limited by the youtube video 
> recording system bit rate?  Also, as a more senior citizen, I believe I 
> am unlikely to be able to hear the difference between the flac and other 
> options.
> 
> I know there are members in this group who have significant knowledge 
> and experience in these matters and I am hoping for suggestions for the 
> best format for my purposes.

Are you using the -x / --extract-audio option with youtube-dl? I have not used 
it for a while, but last time I did, you got either AAC (in a .m4a) or an Opus, 
both perfectly reasonable formats.

It looks like you can pass an option to request m4a audio, which could help if 
you want to play back on an Apple device or an older Android. It should 
otherwise default to what it considers the best available.

https://github.com/ytdl-org/youtube-dl#format-selection-examples

Is there a particular reason you wish to re-encode?

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Demise of iPlayer

[Dave Lambley]

> On 09/18/2020 12:55 AM Chris Brady  wrote:
> 
>  
> As you guys likely know the Beeb is ceasing use of Flash for both
> video and audio (sorry sounds).
> 
> Where does this place get_iplayer ?

IIRC, get_iplayer lost the ability to use Flash some time ago. It is not 
mentioned on https://github.com/get-iplayer/get_iplayer/wiki/modesref

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Which Linux distro with get_iplayer support

[Dave Lambley]

> On May 28, 2020 7:03 PM Dave Widgery  wrote:
> 
>  
> Thanks for all the replies you have given me something to think about,
> sorry if I didn't explain properly why the change, our notebooks are a
> few years old and starting to struggle with all the bloatware that
> comes with windows, I have looked at replacing them and all the cheap
> notebooks now only appear to come with tiny SSD disk drives, I want to
> have a reasonable amount of local disk space but cannot justify
> spending a lot of money on two higher spec PC's, so the next option is
> to use linux, but my wife needs something that looks fairly familiar
> to her hence the look and feel of the windows interface, also most
> programs that we use like LibreOffice are available in native forms
> but we do have a few odd windows programs that aren't available and I
> would rather not duel boot the pc hence the requirement for windows
> emulation.
> 
> Thanks again
> Dave

I've not done this for maybe a decade, but re-installing Windows was something 
we used to do to ageing machines. There is an option buried somewhere on the 
Start Menu to generate you a "Recovery USB stick". (I am amused that Microsoft 
are aware that the software often bundled with new computers has a negative 
value, and can thus charge a premium for "Windows Signature Edition" 
https://www.microsoft.com/en-gb/store/b/laptops )

I suspect you'll find a Linux machine with Windows emulation harder to use than 
one or the other. If you stick to something common like plain Ubuntu or Mint 
(or Windows!) solutions to problems will be easier to find.

Many routers let you connect a USB hard disc and share the contents over your 
network. That may make a cheap SSD laptop more palatable.

All the best,
Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Can get_iplayer still download radio?

[Dave Lambley]

> On May 3, 2020 2:28 PM Owen Smith  wrote:
> 
>  
> That's it, it was originally broadcast in Matrix HJ. As you say, not clear if 
> the rebroadcast is matrix encoded but unless they've re-mastered it in some 
> way it probably is.

I've been admiring the diagrams and matrices in 
http://downloads.bbc.co.uk/rd/pubs/reports/1977-02.pdf

It looks rather phase sensitive. I'd fear that the psychoacoustic compression 
on the iPlayer might harm that, but you never know until you try. You might be 
better off searching for other sources of recordings though.

> 
> My friend develops his own matrix decoding solutions using multiple steps in 
> off the shelf audio processing software, like Audacity. This is not remotely 
> real time, it takes many hours to do one album. But if you're doing matrix 
> decoding you can keep the output and listen to it again. He and I have never 
> understood the general obsession with decoding matrix quad in real time. Do 
> it well and do it once.

Do share your results, I am curious. If the process is similar to the one in 
the paper, it looks like the kind of thing modern CPUs can eat for breakfast, 
if done in something compilable. The block diagrams are reminiscent of audio 
codecs and software defined radio.

I imagine you could do a better job of decoding it than was ever possible in 
1977. The rather cruder Dolby Surround system has been through multiple 
revisions over the years.

All the best,
Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: An OT thread about the forums

[Dave Lambley]

> On April 12, 2020 4:49 PM Roger Bell_West  wrote:
> 
>  
> On Sun, Apr 12, 2020 at 03:32:17PM +0100, SquarePenguin wrote:
> >2) I close the forums to new threads/posts but keep it online.
> >
> >This is great for keeping the resource accessible to the community, it's a
> >PITA for maintenance and ongoing costs (it ain't free to host the forums).
> 
> 2a) possibly in addition to this, start a new forum with better
> software? I don't use the forum because I don't get on with the
> standard interface; but I find that Discourse is remarkably
> non-painful to run.

I was not aware that there even was web forum, but I am painfully aware how 
forums can become unpleasant. The one Discourse forum I am on however is 
remarkably non-toxic.

All the best,
Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: OT Video Conversion

[Dave Lambley]

On Sun, 13 Oct 2019 at 16:26, Dave Widgery  wrote:
>
> Hi
> I would like to copy some of my DVD collection onto hard disk to make
> them easier to access and I am getting very frustrated with
> downloading programs that claim to be free versions only to find out
> that in fact that it is a trial version, or in the case of a
> comprehensive program like Handbrake that is free but takes for ever
> to run on my pc.
> ffmpeg seams to be a very efficient program for converting video, is
> it possible to use it to extract video directly from the DVD video
> files, if so can anyone point me in the direction of a tutorial that
> explains how to do it.
>
> I realise that it is unlikely to work directly from the DVD so I am
> using DVDshrink which allows me to create a hard disk decrypted
> version of the DVD ahead of any other video processing.

I have used the dvdbackup program under Linux to dump the entire DVD.
VLC, and probably other players, will happily play from the resulting
directory structure as if it was the original DVD.

https://wiki.archlinux.org/index.php/dvdbackup#The_whole_DVD

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: no more hslv format ?

[Dave Lambley]

On 5 May 2018 at 00:29, RS <richard...@zoho.com> wrote:
> On 04/05/18 23:50, Dave Lambley wrote:
>
>>
>> They would remove the interlace to bring the framerate to 50fps before
>> encoding. There are many methods of doing so, ffmpeg can do this for
>> example.
>>
> This article
>
> https://superuser.com/questions/253691/how-to-convert-108050i-72050p-using-ffmpeg
>
> says, "Just like interlacing progressive content divides each frame into 2
> fields, thus doubling the frame rate, the common method of deinterlacing is
> to combine each 2 fields into 1 frame, which reduces the rate by 2, thus
> taking 50 into 25fps. You can of course double each final frame, but that
> does not provide any benefit."
>
> Do you disagree?

eg., https://ffmpeg.org/ffmpeg-filters.html#bwdif (and other) filters
can give you good quality 50fps frames, without duplication, from
interlaced 25fps video. I have no idea what the BBC use, but they have
plenty of choice.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: no more hslv format ?

[Dave Lambley]

On 3 May 2018 at 00:24, RS  wrote:
> On 02/05/18 23:21, Owen Smith wrote:
>
>>
>> I've been mystified for a while why people talked about "dropping every
>> other frame" as if it were trivial to do, and an email earlier in this chain
>> looked like someone was trying to do that again. I was explaining why that
>> simply is not possible in the general case.
>>
>
> If I have caused confusion by talking about dropping alternate frames I
> apologise.  I had come across some posts in another forum which suggested it
> could be done, but I now recognise I was wrong.  What makes it worse is that
> I have since come across a thread in this listserver from two years ago
> where I was asking exactly the same questions, and Vangelis pointed out I
> was wrong and directed me to a Wikipedia article on H.264.
>
> It is not possible to change the frame rate using -c:v=copy in ffmpeg; it is
> necessary to re-encode which is why it takes so long.  Inevitably there will
> be losses, added to which the codecs available to us may be inferior to
> those used by the BBC.
>
> I think it was Nick Payne who said he had experimented with re-encoding in
> HEVC (H.265) and found that the file size was the same for 25fps as it was
> for 50fps, which led him to conclude that frames were being duplicated to
> achieve 50fps.
>
> The broadcast signal (at least on satellite) is 1920x1080i at 25fps usually
> with two audio streams, AC3 and NAR.  If the broadcast signal is recorded,
> the resultant file size is about 3GByte/h.
>
> The BBC'S explanation of what it does with the broadcast signal is, "The
> Elemental encoders are used to convert the 1920x1080 interlaced content to
> 960x540 for progressive encoding at 50fps."

Searching turns up this,

http://dpp-assets.s3.amazonaws.com/wp-content/uploads/specs/bbc/TechnicalDeliveryStandardsBBCFile.pdf

... which tells you that 25fps interlaced is what programme makers
will be supplying for HD programmes.

> It also says, "The 50fps, 1280x720 profile, however, will be available to
> those with 5Mbit/s broadband connections." but it does not explain where it
> comes from or why it cannot generate a 1280x720p 25fps profile.

They would remove the interlace to bring the framerate to 50fps before
encoding. There are many methods of doing so, ffmpeg can do this for
example.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Podcast sample rate

[Dave Lambley]

On 17 July 2017 at 17:32, RS <richard...@zoho.com> wrote:
>> From: Dave Lambley
>> Sent: Monday, July 17, 2017 15:43
>
>
>> There are also CODEC internals to consider. The LAME people used to
>> recommend resampling down to 44.1kHz before encoding, because that was
>> the sample rate they had done most of their tuning against.
>
>
> I didn't know about that recommendation and it may be what has influenced
> the BBC.  I have transcoded a fair number of 48kHz sample rate AAC files to
> MP3 using LAME with no audible problems.  That has mainly been at 128kbit/s.
> Maybe problems occur at lower bit rates.  At 48kbit/s Mediainfo shows the
> sampling rate as 48.0kHz / 24.0kHz.  I guess that means a Nyquist frequency
> of 12kHz.  Moneybox used to use a very low rate for its podcast.  I think it
> may have been a sample rate of 16kHz and a bit rate of 32kbit/s.

Surely a Nyquist frequency of 24kHz for a 48kHz sample rate? I don't
remember where I heard the "stick to 44.1" lore from, but Google has
turned up https://hydrogenaud.io/index.php/topic,9105.0.html

Yes, I'd have thought that any problems relating to tuning would
related to the psycho-acoustic stuff, and so would vanish given a high
enough bit rate.

>> For playback, 48kHz files are relatively rare outside video, and receive
>> less testing.
>
>
> A lot of audio is distributed on CD and of course the CD sampling rate is
> 44.1kHz.
>
>> My old Sony Ericsson phone, for example, would play back
>> 48kHz AAC with periodic digital splats.
>
>
> Was that a 48kHz problem or an AAC problem?  Some players are very fussy
> about the AAC they will play.  I used to have a DAB+ radio which would also
> record DAB+ stations in AAC.  It was a struggle to find a DAB+ station but I
> found one in Poland.  Playback of its own AAC recordings was fine.  It was
> also fine playng MP3 files from its microSD card.  When it played external
> AAC files there was warbling sound over the top.

I can't answer that as the phone sadly died. The files would have been
from get_iplayer in ~2010.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Podcast sample rate

[Dave Lambley]

On 17 July 2017 at 14:05, Jim web  wrote:
> The main problem with 48k -> 44.1k resampling is that it will risk adding
> artifacts. If you're wanting to save bits it would make more sense to tell
> a lossy encoder to simply ignore anything above a given HF limit. (Various
> BBC outlets used to do this some years ago.) The result can then remain 48k
> rate but have the allocated bits used for more detail at lower frequencies.

There are also CODEC internals to consider. The LAME people used to
recommend resampling down to 44.1kHz before encoding, because that was
the sample rate they had done most of their tuning against. For
playback, 48kHz files are relatively rare outside video, and receive
less testing. My old Sony Ericsson phone, for example, would play back
48kHz AAC with periodic digital splats.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: How good is HD supposed to be?

[Dave Lambley]

On 29 April 2016 at 00:57, Dave Liquorice  wrote:
> On Thu, 28 Apr 2016 07:07:33 -0500, artisticforge . wrote:
>
>
 NB hvfhd DOES NOT OFFER HIGHER RESOLUTION (CLARITY), only doubled
 framerate (25FPS x2), which results in smoother scenes where motion is
 involved!
>>>
>>> How does repeating frames improve smoothness of movement? Or does this
>>> encode upscale each field(*) and encode that to increase the temporal
>>> resolution?
>>
>> It is the frames per second that provide the human eye with the
>> persistence of vision, the illusion of motion.
>
> I could show you 100 fps but if there where only 4 different images
> displayed the illusion of motion would be no smoother than 25 fps. You only
> get smoother movement by increasing the number of different images
> displayed.
>
> So if this hvfhd only repeats each frame to get a higher frame there is no
> increase in smoothness. How ever if they take each field, upscale it and
> encode as a frame that would inrease the smoothness.

I believe the frame repeating idea is a red herring. Real 50 frame/s
computer video is a thing which exists. If your browser's up to it you
can see for yourself here,
https://www.youtube.com/watch?v=PmNapQdWFKg

You'll need to choose one of the "p50" resolutions on the Quality menu.

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: How good is HD supposed to be?

[Dave Lambley]

On 28 April 2016 at 10:41, Dave Liquorice  wrote:
>
> On Thu, 28 Apr 2016 03:08:38 +0300, Vangelis forthnet wrote:
>
> >> In addition a high bit-rate (8000kbps) 1920x1080, interlaced at 25fps,
> >> full HD encode is generated, but has not yet been made available.
> >
> > but it has not yet been released publicly...
> > At ca. 8Mps, those will be really huge files!
>
> 3.2 GB/hour roughly. The current 2.5 Mbps (ish) uses about 1 GB/hr.
>
> > NB hvfhd DOES NOT OFFER HIGHER RESOLUTION (CLARITY), only doubled
> > framerate (25FPS x2), which results in smoother scenes where motion is
> > involved!
>
> How does repeating frames improve smoothness of movement? Or does this
> encode upscale each field(*) and encode that to increase the temporal
> resolution?
>
> (*) If good ole 25 frames/50 fields per second has any relevance in the
> digital world.

The BBC could well have non-interlaced 50 frames/s master copies, and
for when they don't there are de-interlacing algorithms which can give
smooth 50 fields/s output from interlaced 25 frames/s video. (I
believe MPlayer has some.)

Dave

___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: 4GB filesize limit

[Dave Lambley]

It might be worth trying this recording on a machine running a 64 bit
build of rtmpdump.  Probably on 64 bit Linux.

You may have trouble playing the files after recording them however.
Some file formats, such as WAV, MP4 and ZIP have a lesser implemented
variant for files greater than 4 GB.

Cheers,
Dave

On 13 February 2014 22:07, Dave Lambley d...@lambley.me.uk wrote:
 It might be worth trying this recording on a machine running a 64 bit build
 of rtmpdump.  Probably on 64 bit Linux.

 You may have trouble playing the files after recording them.  Some file
 formats, such as WAV, MP4 and ZIP have a lesser implemented variant for
 files greater than 4 GB.

 Cheers,
 Dave


 On 13 February 2014 21:00, Paul Phillips paulphillipsdidsb...@gmail.com
 wrote:

 Try this maybe

 Go into command line in Windows and try this - note I've not tested it
 (it's running now, but it will take a while)

 C:\Program Files (x86)\get_iplayer--get 1231 --modes=best --stop
 3:30:00 --attempts 1 --force

 where 1231 is the PID for the ceremony on my PC.

 The above is an attempt to record the first 3.5 hours of the ceremony.
 There was a lot BBC commentary at the end which took it to 4 hours.

 You've got about 15 hours to sort this out before it goes for good from
 iplayer.

 On 13 February 2014 19:35, Peter S Kirk peter.k...@isauk.biz wrote:
  On 13 Feb 2014 at 8:42, Derek Balling Derek Balling dr...@megacity.org
  wrote:
 
  What OS are you on?
 
  4GB file limit sounds a lot more like an OS-/Filesystem-driven
  limitation than a get_iplayer limitation.
 
  D
 
 
  On Feb 13, 2014, at 8:35 AM, Sam P haj...@hotmail.com wrote:
 
   i just tried to record with winter olympics opening ceremony 720p:
 
  http://www.bbc.co.uk/iplayer/episode/b03tvhy0/hd/Winter_Olympics_Sochi_2014_Opening_Ceremony/
  
   Unfortunately the recording stops at 3.99GB. Please fix this issue
   and allow us to download the full file or
  atleast
  split it to seperate 4gb files.
  
   i'm using v2.85 btw.
  
 
  As dinky has pointed out before, it appears to be an rtmpdump issue.
  See:
  https://code.google.com/p/get-iplayer-automator/issues/detail?id=84
 
  Cheers,
 
  Peter
 
  15GB + 5GB bonus Free cloud storage, no credit card details required:
  https://copy.com?r=VnqHxA - 5GB bonus for both for using referal link
 
  Cloud storage space and app provided by https://www.barracuda.com
 
 
  ___
  get_iplayer mailing list
  get_iplayer@lists.infradead.org
  http://lists.infradead.org/mailman/listinfo/get_iplayer



 --
 Paul Phillips

 ___
 get_iplayer mailing list
 get_iplayer@lists.infradead.org
 http://lists.infradead.org/mailman/listinfo/get_iplayer



___
get_iplayer mailing list
get_iplayer@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/get_iplayer

Re: Invalid XML Entities included in metadata file

[Dave Lambley]

On 22/03/13 18:36, Roger Bell_West wrote:

On Fri, Mar 22, 2013 at 06:17:23PM +, Ian W Taylor wrote:

I think the problem is that get_iplayer uses HTML encode_entities()
and there are about 250 entities defined in HTML but only 5 in the
XML specification.  I've read that XML just defines quot; amp;
apos; lt; and gt; for the ' characters.  However the generic
metadata XML file includes things like pound; for the British? Pound
sign in the description nodes.


I've had this problem too, and would like not to have to sanitise the
XML before reading it.


Perhaps the problem is that a DTD used to be obtained from web site
named in the 2nd line of the XML, but is no longer available ?
program_meta_data
xmlns=http://linuxcentre.net/xmlstuff/get_iplayer; revision=1


No, nothing specifically has to be at that URL; it's just a unique
identifier for the namespace.


Try the attached patch, which switches to numeric entity encoding for XML.

Regards,
Dave
From 01ee0e8ecf40115fdb8bfad1b237449cfc2160df Mon Sep 17 00:00:00 2001
From: Dave Lambley g...@davel.me.uk
Date: Fri, 22 Mar 2013 19:58:20 +
Subject: [PATCH] Use XML safe numeric entities.

---
 get_iplayer |   30 +++---
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/get_iplayer b/get_iplayer
index ff8b480..f210317 100755
--- a/get_iplayer
+++ b/get_iplayer
@@ -56,7 +56,7 @@ use File::Path;
 use File::stat;
 use File::Spec;
 use Getopt::Long;
-use HTML::Entities;
+use HTML::Entities qw/ encode_entities_numeric /;
 use HTTP::Cookies;
 use HTTP::Headers;
 use IO::Seekable;
@@ -2231,15 +2231,15 @@ sub create_xml {
 			$program_count{ $this-{name} }++;
 		}
 		for my $name ( sort keys %program_index ) {
-			print XML \t\tcontainer title=\.encode_entities( $name ). ($program_count{$name})\\n if $opt-{fxd};
+			print XML \t\tcontainer title=\.encode_entities_numeric( $name ). ($program_count{$name})\\n if $opt-{fxd};
 			print XML \tStreams\n if $opt-{mythtv};
-			print XML \t\tName.encode_entities( $name )./Name\n if $opt-{mythtv};
+			print XML \t\tName.encode_entities_numeric( $name )./Name\n if $opt-{mythtv};
 			for my $this (@_) {
 my $pid = $this-{pid};
 # loop through and find matches for each progname
 if ( $this-{name} eq $name ) {
-	my $episode = encode_entities( $this-{episode} );
-	my $desc = encode_entities( $this-{desc} );
+	my $episode = encode_entities_numeric( $this-{episode} );
+	my $desc = encode_entities_numeric( $this-{desc} );
 	my $title = ${episode};
 	$title .=  ($this-{available}) if $this-{available} !~ /^(unknown|)$/i;
 	if ( $opt-{fxd} ) {
@@ -2284,10 +2284,10 @@ sub create_xml {
 			push @{ $channels{ $this-{channel} } }, $this-{name};
 		}
 		for my $channel ( sort keys %channels ) {
-			print XML \t\tcontainer title=\.encode_entities( $channel ).\\n if $opt-{fxd};
+			print XML \t\tcontainer title=\.encode_entities_numeric( $channel ).\\n if $opt-{fxd};
 			print XML
 \tFeed\n.
-\t\tName.encode_entities( $channel )./Name\n.
+\t\tName.encode_entities_numeric( $channel )./Name\n.
 \t\tProviderBBC/Provider\n.
 \t\tStreams\n if $opt-{mythtv};
 			for my $name ( sort keys %program_index ) {
@@ -2297,14 +2297,14 @@ sub create_xml {
 	$match = 1 if $_ eq $name;
 }
 if ( $match ) {
-	print XML \t\t\tcontainer title=\.encode_entities( $name ). ($program_count{$name})\\n if $opt-{fxd};
+	print XML \t\t\tcontainer title=\.encode_entities_numeric( $name ). ($program_count{$name})\\n if $opt-{fxd};
 	#print XML \t\tStream\n if $opt-{mythtv};
 	for my $this (@_) {
 		# loop through and find matches for each progname for this channel
 		my $pid = $this-{pid};
 		if ( $this-{channel} eq $channel  $this-{name} eq $name ) {
-			my $episode = encode_entities( $this-{episode} );
-			my $desc = encode_entities( $this-{desc} );
+			my $episode = encode_entities_numeric( $this-{episode} );
+			my $desc = encode_entities_numeric( $this-{desc} );
 			my $title = ${episode} ($this-{available});
 			if ( $opt-{fxd} ) {
 print XML
@@ -2319,7 +2319,7 @@ sub create_xml {
 			} elsif ( $opt-{mythtv} ) {
 print XML 
 	\t\t\tStream\n.
-	\t\t\t\tName.encode_entities( $name )./Name\n.
+	\t\t\t\tName.encode_entities_numeric( $name )./Name\n.
 	\t\t\t\tindex$this-{index}/index\n.
 	\t\t\t\ttype$this-{type}/type\n.
 	\t\t\t\tUrl${pid}.mov/Url\n.
@@ -2359,9 +2359,9 @@ sub create_xml {
 			print XML \t\tcontainer title=\$folder\\n;
 			for my $this (@_) {
 my $pid = $this-{pid};
-my $name = encode_entities( $this-{name} );
-my $episode = encode_entities( $this-{episode} );
-my $desc = encode_entities( $this-{desc} );
+my $name = encode_entities_numeric( $this-{name} );
+my $episode = encode_entities_numeric( $this-{episode} );
+my $desc = encode_entities_numeric( $this-{desc} );
 my $title = ${name