On Wed, Oct 31, 2012 at 09:43:42AM -0500, Romain Beauxis wrote: > Hi, > > 2012/10/31 Ken Restivo <[email protected]>: > > On Tue, Oct 30, 2012 at 09:54:36AM -0500, Romain Beauxis wrote: > >> Hi Ken! > >> > >> 2012/10/29 Ken Restivo <[email protected]>: > >> > Hello folks, thanks for liquidsoap. I'm very glad someone created a > >> > streaming engine written in a functional language. > >> > >> Thanks! > >> > >> > Earlier this summer I set up a server for a micropower radio station in > >> > a remote location. > >> > > >> > It's running an old 3Ghz P4 with 2GB RAM, Ubuntu 10.04, and airtime. > >> > > >> > They wanted a jukebox for times there was no scheduled show playing, so, > >> > rather than tweak airtime, I set up a separate liquidsoap instance that > >> > ran a custom script. The script is below. > >> > > >> > It basically plays a jukebox, but falls forward to a live stream or to > >> > airtime, if either of those become available. It worked for a month or > >> > two flawlessly. > >> > > >> > The problem is, after a few months running, that it'd periodically > >> > stutter approximately 400ms of audio endlessly, then after 10-60 seconds > >> > move on to another frame, stutter that one, then it'd do a > >> > scrambled-sounding insane thing that sounds like an mp3 stream seeking > >> > past its key frame, then go back to stuttering, for hours. The problem > >> > would clear itself after some hours had passed. CPU usage was minimal at > >> > those times. There's no RT kernel, but I had liquidsiap nice'd -20 so it > >> > ran with higher priority than anything. The problem seems to happen > >> > randomly, intermittendly, typically once or twice a day, at no > >> > predictable time, and regardless of whether the active source is the > >> > jukebox, the livestream, or the airtime stream. > >> > > >> > The logs all look completely normal, no errors, no overruns, etc. At one > >> > point I did see an overrun, at the point where the error STOPPED > >> > happening, not when it started, which was odd. At the time that it's > >> > stuttering, the jukebox (or livestream, or airtime), looks like it's > >> > just cruising along doing what it's supposed to be doing. > >> > > >> > This was with Ubuntu's (old) liquidsoap package. I built a custom > >> > liquidsoap-1.0.1 from source, installed it, and ran that.... and it did > >> > not solve the problem. > >> > > >> > I've since ditched the separate liquidsoap instance completely, and now > >> > run these functions from inside airtime. It did not solve the problem. > >> > > >> > I asked on the linux audio list, and one of the ALSA devs pointed out > >> > that stuttering is caused by the application not sending data to the > >> > driver, so the hardware just keeps playing the same frame over and over > >> > again until it gets a new one from the application. But this isn't one > >> > frame (~40ms) it's a 400ms chunk. And the error indicates something more > >> > complex. > >> > > >> > Since this is airtime, I'm certain I'll get at least one reply telling > >> > me to go ask the Airtime people. They, in turn, sent me here. I did just > >> > today upgrade to airtime 2.2 which uses liquidsoap 1.0.1, but I didn't > >> > see any issues related to this problem, or anything in the changelog > >> > indicating it fixed it. Though, they are now using mp3replaygain in this > >> > new version, so CPU usage has gone sky high. I'll know by tomorrow if > >> > the upgrade fixed anything. > >> > > >> > My questions are: > >> > > >> > 1) Are there any things in the below script that look like the could > >> > cause liquidsoap to do this crazy thing? I am not new to functional > >> > programming or linux audio, but I am new to liquidsoap. Am I doing > >> > anything obviously wrong in the code below? > >> > > >> > 2) How would I find out when is happening, or that it had happened? I > >> > found the rms() command, which looks like it could be helpful for > >> > detecting silence. But what unit of sound is this 400ms chunk? What cuts > >> > it up into those size hunks? Is there some way to compare whether two or > >> > more sequential blocks like that are exactly equal to each other (and > >> > flag it perhaps as an error)? > >> > > >> > 3) How could I get more data as to what exactly is wrong? > >> > >> Logs could help, particularly in verbose mode -- set("log.level",4) > >> > > > > Thanks, I didn't think to increase log verbosity, doh. Could try that. > > > >> As for your issue, I would suspect some bug with the alsa > >> driver/output. First thing to try to confirm that would be to use an > >> alternative output. I'd suggest output.ao which is generally the most > >> robust. > > > > We tried it with AO. Same problem. > > > > I should also note that I misspoke in my original report: at the time the > > stuttering would happen, liquidsoap would inded be at minimal 5% load, but > > the rest of the system was quite loaded down! Apache would be at 70% load > > or something similarly ridiculous, for short periods, and average about 40% > > CPU. So, perhaps this is just resource starvation? > > Sounds like that's what it is! > > >> > >> If output.ao works and you want to help us fixing the bug, even though > >> your jukebox should be now working, then we could try to see what > > > > > > The jukebox is not working. Interestingly, we tried just straight stock > > airtime, removing my jukebox modifications completely, and... same exact > > problem. So it's not my script causing it. > > > > HOWEVER. > > > > We've been running mplayer on a live http mp3 stream for 24 hours now, no > > glitches, no problems. But also, no load. So the problem is not ALSA. It's > > liquidsoap, or, perhaps liquidsoap + load. > > If by running mplayer you mean decoding a stream, then in general > decoding is much less CPU intensive than encoding. Since liquidsoap > also encodes, that will make a significant different with using > mplayer.. > > > Another question: what's the significance of the 400ms stutter period? It's > > very consistent. Is that the size of an mp3 frame? The length of a > > liquidsoap buffer? Is it the interval of a poll or select somewhere? Or a > > system timer of some kind? > > It most likely is liquidsoap's internal audio frame duration.. >
Thanks, that makes sense. > > Finally, how well does liquidsoap run under RT priority? I've used Ingo > > kernels for 6 years now, perhaps it might be worthwhile to try installing > > one on this box and giving liquidsoap rt prio > > I have no idea how liquidsoap works with RT. Would be nice to try and > report here! > > >> audio card you are using. However, debugging alsa on a card that we do > >> not own is kinda tricky.. You may also want to test with or without > >> the bufferize option. > >> > > > > It was screwing up using an HDA-INTEL card. So they put in an ES1371/1 card > > and tried with that... same problem. > > > > We also tried moving from the build in ethernet card to a PCI card, and > > then scooting the cards around so that the network was on its own > > interrupt. Also, same problem. > > Yeah no, I think you have a load problem. Check out your logs, you > should see "Catchup" messages.. > zgrep Catchup /var/log/airtime/pypo-liquidsoap/*gz comes up with nothing. Also, no buffer overrun/underrun messages either. Sure looks like CPU starvation though. Thanks again for your help on this! -ken ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct _______________________________________________ Savonet-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/savonet-users
