*I should also note that I misspoke in my original report: at the time the
stuttering would happen, liquidsoap would inded be at minimal 5% load, but
the rest of the system was quite loaded down! Apache would be at 70% load
or something similarly ridiculous, for short periods, and average about 40%
CPU. So, perhaps this is just resource starvation?*
*
*
Sounds like it. Is the server dedicated to Airtime or is it hosting other
sites? If it is only hosting Airtime, then perhaps there is a bug in the
PHP code which causes high CPU load, although we haven't seen this yet.
Anything in the apache error logs?
On Wed, Oct 31, 2012 at 3:19 AM, Ken Restivo <[email protected]> wrote:
> On Tue, Oct 30, 2012 at 09:54:36AM -0500, Romain Beauxis wrote:
> > Hi Ken!
> >
> > 2012/10/29 Ken Restivo <[email protected]>:
> > > Hello folks, thanks for liquidsoap. I'm very glad someone created a
> streaming engine written in a functional language.
> >
> > Thanks!
> >
> > > Earlier this summer I set up a server for a micropower radio station
> in a remote location.
> > >
> > > It's running an old 3Ghz P4 with 2GB RAM, Ubuntu 10.04, and airtime.
> > >
> > > They wanted a jukebox for times there was no scheduled show playing,
> so, rather than tweak airtime, I set up a separate liquidsoap instance that
> ran a custom script. The script is below.
> > >
> > > It basically plays a jukebox, but falls forward to a live stream or to
> airtime, if either of those become available. It worked for a month or two
> flawlessly.
> > >
> > > The problem is, after a few months running, that it'd periodically
> stutter approximately 400ms of audio endlessly, then after 10-60 seconds
> move on to another frame, stutter that one, then it'd do a
> scrambled-sounding insane thing that sounds like an mp3 stream seeking past
> its key frame, then go back to stuttering, for hours. The problem would
> clear itself after some hours had passed. CPU usage was minimal at those
> times. There's no RT kernel, but I had liquidsiap nice'd -20 so it ran with
> higher priority than anything. The problem seems to happen randomly,
> intermittendly, typically once or twice a day, at no predictable time, and
> regardless of whether the active source is the jukebox, the livestream, or
> the airtime stream.
> > >
> > > The logs all look completely normal, no errors, no overruns, etc. At
> one point I did see an overrun, at the point where the error STOPPED
> happening, not when it started, which was odd. At the time that it's
> stuttering, the jukebox (or livestream, or airtime), looks like it's just
> cruising along doing what it's supposed to be doing.
> > >
> > > This was with Ubuntu's (old) liquidsoap package. I built a custom
> liquidsoap-1.0.1 from source, installed it, and ran that.... and it did not
> solve the problem.
> > >
> > > I've since ditched the separate liquidsoap instance completely, and
> now run these functions from inside airtime. It did not solve the problem.
> > >
> > > I asked on the linux audio list, and one of the ALSA devs pointed out
> that stuttering is caused by the application not sending data to the
> driver, so the hardware just keeps playing the same frame over and over
> again until it gets a new one from the application. But this isn't one
> frame (~40ms) it's a 400ms chunk. And the error indicates something more
> complex.
> > >
> > > Since this is airtime, I'm certain I'll get at least one reply telling
> me to go ask the Airtime people. They, in turn, sent me here. I did just
> today upgrade to airtime 2.2 which uses liquidsoap 1.0.1, but I didn't see
> any issues related to this problem, or anything in the changelog indicating
> it fixed it. Though, they are now using mp3replaygain in this new version,
> so CPU usage has gone sky high. I'll know by tomorrow if the upgrade fixed
> anything.
> > >
> > > My questions are:
> > >
> > > 1) Are there any things in the below script that look like the could
> cause liquidsoap to do this crazy thing? I am not new to functional
> programming or linux audio, but I am new to liquidsoap. Am I doing anything
> obviously wrong in the code below?
> > >
> > > 2) How would I find out when is happening, or that it had happened? I
> found the rms() command, which looks like it could be helpful for detecting
> silence. But what unit of sound is this 400ms chunk? What cuts it up into
> those size hunks? Is there some way to compare whether two or more
> sequential blocks like that are exactly equal to each other (and flag it
> perhaps as an error)?
> > >
> > > 3) How could I get more data as to what exactly is wrong?
> >
> > Logs could help, particularly in verbose mode -- set("log.level",4)
> >
>
> Thanks, I didn't think to increase log verbosity, doh. Could try that.
>
> > As for your issue, I would suspect some bug with the alsa
> > driver/output. First thing to try to confirm that would be to use an
> > alternative output. I'd suggest output.ao which is generally the most
> > robust.
>
> We tried it with AO. Same problem.
>
> I should also note that I misspoke in my original report: at the time the
> stuttering would happen, liquidsoap would inded be at minimal 5% load, but
> the rest of the system was quite loaded down! Apache would be at 70% load
> or something similarly ridiculous, for short periods, and average about 40%
> CPU. So, perhaps this is just resource starvation?
>
> >
> > If output.ao works and you want to help us fixing the bug, even though
> > your jukebox should be now working, then we could try to see what
>
>
> The jukebox is not working. Interestingly, we tried just straight stock
> airtime, removing my jukebox modifications completely, and... same exact
> problem. So it's not my script causing it.
>
> HOWEVER.
>
> We've been running mplayer on a live http mp3 stream for 24 hours now, no
> glitches, no problems. But also, no load. So the problem is not ALSA. It's
> liquidsoap, or, perhaps liquidsoap + load.
>
> Another question: what's the significance of the 400ms stutter period?
> It's very consistent. Is that the size of an mp3 frame? The length of a
> liquidsoap buffer? Is it the interval of a poll or select somewhere? Or a
> system timer of some kind?
>
> Finally, how well does liquidsoap run under RT priority? I've used Ingo
> kernels for 6 years now, perhaps it might be worthwhile to try installing
> one on this box and giving liquidsoap rt prio
>
>
> > audio card you are using. However, debugging alsa on a card that we do
> > not own is kinda tricky.. You may also want to test with or without
> > the bufferize option.
> >
>
> It was screwing up using an HDA-INTEL card. So they put in an ES1371/1
> card and tried with that... same problem.
>
> We also tried moving from the build in ethernet card to a PCI card, and
> then scooting the cards around so that the network was on its own
> interrupt. Also, same problem.
>
>
> -ken
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Savonet-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/savonet-users
>
--
Open source radio in the cloud. Get yours now! ---> http://airtime.pro
Martin Konecny
Software Developer, Sourcefabric
[email protected]
720 Bathurst St. Suite 203
M5S 2R4, Toronto, ON, Canada
+1 (416) 892-8420 (Cell)
Skype: martin.konecny15
http://www.sourcefabric.org
http://www.twitter.com/Sourcefabric
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Savonet-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/savonet-users
