Hi all, I think the open pipes point to the problem. Beside external encoders, calling external processes may also open pipes and it's possible that that would be the issue.
I see that you are using an external command for your php bridge. Can you test without it and, if so, do you observe the same amount of open pipes? Also, same remark for any external process that you may be calling in your script. Romain 2014-01-30 Alexander <truenr...@gmail.com>: > Hello! > > No, we don't use external encoder. But, as we support mp3/ogg/aac with big > variety of bitrates, we have a lot of definitions for each format. I will > send library we use to create/remove outputs (we allow to change output > formats/bitrates on the fly) in a separate main directly to you, just in > case. > > To be honest, sometimes i use "kill $pid" to stop some liquidsoap processes, > but i don't use "-9" parameter. As i understand, there is a difference > between "kill $pid" and "kill -9 $pid" (in first case process is able to > close all connections, remove temp files, etc). But it is _very_ rare. Usual > process of shutting station down liquidsoap is to execute server command via > socket, which executes "shutdown()" function to perform clean shutdown. > > But, i have checked what happens when i execute "kill $pid". I have removed > all files from /tmp, started all liquidsoap processes and then killed them > with "kill $pid1 $pid2 $pid3". All callbacks are executed correctly. All > files from /tmp are removed. Everything looks fine. > > I can still see 344 mp3 files in /tmp folder. But i am 100% sure those are > files, that were left after liquidsoap processes had crashed. > > Also, i can't see any reports about dead stations after yesterday's changes > (i have commented out cue cuts). But we shoud keep in mind that i have > rebooted all processes yesterday. Perhaps collapse time just have not come > (see report about "ls -l /proc/$pid/fd" below). > > I tried to execute server command request.alive on several stations. i can > still see 3 requests per station. > > I executed "ls -l /proc/$pid/fd" for several stations, picture is quite > different. I will give output of "ls -l /proc/$pid/fd | wc -l" (as a > summary) for all stations: 393, 374, 28, 27, 48, 411, 400, 321, 109, 301, > 331, 371, 28, 337, 37 > > Hope you will find this useful. > > > > 2014-01-30 David Baelde <david.bae...@gmail.com>: > >> Hi Alexander, >> >> Thanks for the extra info. Nothing to worry about regarding liquidget, >> it's a harmless wrapper around wget. The logs are useful, as they >> point to pipes rather than downloaded files. You must be using some >> feature (external encoder, maybe) that uses pipes for interprocess >> communication, and that does not close them properly. That being said, >> 372 temporary files is still a lot, but that must be a different >> problem... do you kill -9 liquidsoap instances, by any chance? >> >> Cheers, >> >> David >> >> On Wed, Jan 29, 2014 at 12:46 PM, Alexander <truenr...@gmail.com> wrote: >> > Hello! >> > I've also added your email directly to recipients list (in case >> > sourcefourge >> > lists will crop attachments). Please find answers on your questions >> > below: >> > >> >> Are you using the standard resolver for http, that is wget? >> > Hmm.. i suppose that we use wget. ./configure script shows in table >> > following data (and i have wget installed): >> > * Additional libraries >> > - wget URI resolver : requires wget at runtime >> > >> > But, in liquidsoap logs i found following line: '[protocols.external:3] >> > Found "/usr/local/lib/liquidsoap/scm/liquidget"'. So, i am not sure. >> > >> > >> >> Do you have a reasonable number of alive requests? You can see this >> >> using >> >> the "request.alive" server command. On the server you can also perform >> >> "request.trace" to see what happened to a request. >> > I had to enable telnet and to restart stations (usually we use only >> > sockets). So, i am not sure it would be informative enough. Anyway, i am >> > attaching tcp.log file with server commands execution log. >> > >> > >> >> Can you identify the too many files in "ls -l /proc/<liq pid>/fd"? If >> >> they >> >> are the downloaded files, you should see temporary file names >> > starting with "liq". It would be useful to check if these files are all >> > the >> > downloaded files so far or only some. >> > I see only one downloaded file there. All others are pipes (oh god, how >> > many >> > of them i can see here!). Please find "station-11-fd.log" file attached >> > with >> > output for 1 liquidsoap process. >> > >> > One suggestion: as liquidsoap puts all files in /tmp folder, i just >> > checked >> > how many files are there. 372 audio files (that are certainly donwloaded >> > via >> > liquidsoap). It should be too many files for 15 liquidsoap processes >> > working >> > at one time. >> > >> > >> >> Just to be sure, can you reproduce the issue without cue_cut? >> > Of course, i have disabled it and restarted all stations, i will >> > continue >> > reporting if liquidsoap processes will die again. >> > >> > >> > Hope this information helps. Thanks for your response! >> > >> > >> > 2014-01-29 David Baelde <david.bae...@gmail.com> >> >> >> >> Hi, >> >> >> >> No clue so far, so I'll ask a few questions instead. >> >> >> >> * Are you using the standard resolver for http, that is wget? >> >> >> >> * Do you have a reasonable number of alive requests? You can see this >> >> using the "request.alive" server command. On the server you can also >> >> perform "request.trace" to see what happened to a request. >> >> >> >> * Can you identify the too many files in "ls -l /proc/<liq pid>/fd"? >> >> If they are the downloaded files, you should see temporary file names >> >> starting with "liq". It would be useful to check if these files are >> >> all the downloaded files so far or only some. >> >> >> >> * Just to be sure, can you reproduce the issue without cue_cut? >> >> >> >> Thanks, >> >> >> >> David >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> WatchGuard Dimension instantly turns raw network data into actionable >> >> security intelligence. It gives you real-time visual feedback on key >> >> security issues and trends. Skip the complicated setup - simply import >> >> a virtual appliance and go from zero to informed in seconds. >> >> >> >> >> >> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk >> >> _______________________________________________ >> >> Savonet-users mailing list >> >> Savonet-users@lists.sourceforge.net >> >> https://lists.sourceforge.net/lists/listinfo/savonet-users >> > >> > >> > >> > >> > -- >> > Kind regards, >> > Alexander. >> >> >> >> -- >> David > > > > > -- > Kind regards, > Alexander. > > ------------------------------------------------------------------------------ > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you real-time visual feedback on key > security issues and trends. Skip the complicated setup - simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Savonet-users mailing list > Savonet-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/savonet-users > ------------------------------------------------------------------------------ WatchGuard Dimension instantly turns raw network data into actionable security intelligence. It gives you real-time visual feedback on key security issues and trends. Skip the complicated setup - simply import a virtual appliance and go from zero to informed in seconds. http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk _______________________________________________ Savonet-users mailing list Savonet-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/savonet-users
