Hello!
After 2 or 3 days uptime liquidsoap processes started to crash again :((
Error messages in log seems to be same.
Hoping for a miracle :)
2014-01-31 David Baelde <david.bae...@gmail.com>:
> Hi guys,
>
> Just a thought for Romain: how about pipes used as locks inside duppy?
> Could there be a feature here that makes specific use of them?
>
> Cheers,
>
> David
>
> On Thu, Jan 30, 2014 at 5:26 PM, Romain Beauxis <to...@rastageeks.org>
> wrote:
> > Hi all,
> >
> > I think the open pipes point to the problem. Beside external encoders,
> > calling external processes may also open pipes and it's possible that
> > that would be the issue.
> >
> > I see that you are using an external command for your php bridge. Can
> > you test without it and, if so, do you observe the same amount of open
> > pipes? Also, same remark for any external process that you may be
> > calling in your script.
> >
> > Romain
> >
> > 2014-01-30 Alexander <truenr...@gmail.com>:
> >> Hello!
> >>
> >> No, we don't use external encoder. But, as we support mp3/ogg/aac with
> big
> >> variety of bitrates, we have a lot of definitions for each format. I
> will
> >> send library we use to create/remove outputs (we allow to change output
> >> formats/bitrates on the fly) in a separate main directly to you, just in
> >> case.
> >>
> >> To be honest, sometimes i use "kill $pid" to stop some liquidsoap
> processes,
> >> but i don't use "-9" parameter. As i understand, there is a difference
> >> between "kill $pid" and "kill -9 $pid" (in first case process is able to
> >> close all connections, remove temp files, etc). But it is _very_ rare.
> Usual
> >> process of shutting station down liquidsoap is to execute server
> command via
> >> socket, which executes "shutdown()" function to perform clean shutdown.
> >>
> >> But, i have checked what happens when i execute "kill $pid". I have
> removed
> >> all files from /tmp, started all liquidsoap processes and then killed
> them
> >> with "kill $pid1 $pid2 $pid3". All callbacks are executed correctly. All
> >> files from /tmp are removed. Everything looks fine.
> >>
> >> I can still see 344 mp3 files in /tmp folder. But i am 100% sure those
> are
> >> files, that were left after liquidsoap processes had crashed.
> >>
> >> Also, i can't see any reports about dead stations after yesterday's
> changes
> >> (i have commented out cue cuts). But we shoud keep in mind that i have
> >> rebooted all processes yesterday. Perhaps collapse time just have not
> come
> >> (see report about "ls -l /proc/$pid/fd" below).
> >>
> >> I tried to execute server command request.alive on several stations. i
> can
> >> still see 3 requests per station.
> >>
> >> I executed "ls -l /proc/$pid/fd" for several stations, picture is quite
> >> different. I will give output of "ls -l /proc/$pid/fd | wc -l" (as a
> >> summary) for all stations: 393, 374, 28, 27, 48, 411, 400, 321, 109,
> 301,
> >> 331, 371, 28, 337, 37
> >>
> >> Hope you will find this useful.
> >>
> >>
> >>
> >> 2014-01-30 David Baelde <david.bae...@gmail.com>:
> >>
> >>> Hi Alexander,
> >>>
> >>> Thanks for the extra info. Nothing to worry about regarding liquidget,
> >>> it's a harmless wrapper around wget. The logs are useful, as they
> >>> point to pipes rather than downloaded files. You must be using some
> >>> feature (external encoder, maybe) that uses pipes for interprocess
> >>> communication, and that does not close them properly. That being said,
> >>> 372 temporary files is still a lot, but that must be a different
> >>> problem... do you kill -9 liquidsoap instances, by any chance?
> >>>
> >>> Cheers,
> >>>
> >>> David
> >>>
> >>> On Wed, Jan 29, 2014 at 12:46 PM, Alexander <truenr...@gmail.com>
> wrote:
> >>> > Hello!
> >>> > I've also added your email directly to recipients list (in case
> >>> > sourcefourge
> >>> > lists will crop attachments). Please find answers on your questions
> >>> > below:
> >>> >
> >>> >> Are you using the standard resolver for http, that is wget?
> >>> > Hmm.. i suppose that we use wget. ./configure script shows in table
> >>> > following data (and i have wget installed):
> >>> > * Additional libraries
> >>> > - wget URI resolver : requires wget at runtime
> >>> >
> >>> > But, in liquidsoap logs i found following line:
> '[protocols.external:3]
> >>> > Found "/usr/local/lib/liquidsoap/scm/liquidget"'. So, i am not sure.
> >>> >
> >>> >
> >>> >> Do you have a reasonable number of alive requests? You can see this
> >>> >> using
> >>> >> the "request.alive" server command. On the server you can also
> perform
> >>> >> "request.trace" to see what happened to a request.
> >>> > I had to enable telnet and to restart stations (usually we use only
> >>> > sockets). So, i am not sure it would be informative enough. Anyway,
> i am
> >>> > attaching tcp.log file with server commands execution log.
> >>> >
> >>> >
> >>> >> Can you identify the too many files in "ls -l /proc/<liq pid>/fd"?
> If
> >>> >> they
> >>> >> are the downloaded files, you should see temporary file names
> >>> > starting with "liq". It would be useful to check if these files are
> all
> >>> > the
> >>> > downloaded files so far or only some.
> >>> > I see only one downloaded file there. All others are pipes (oh god,
> how
> >>> > many
> >>> > of them i can see here!). Please find "station-11-fd.log" file
> attached
> >>> > with
> >>> > output for 1 liquidsoap process.
> >>> >
> >>> > One suggestion: as liquidsoap puts all files in /tmp folder, i just
> >>> > checked
> >>> > how many files are there. 372 audio files (that are certainly
> donwloaded
> >>> > via
> >>> > liquidsoap). It should be too many files for 15 liquidsoap processes
> >>> > working
> >>> > at one time.
> >>> >
> >>> >
> >>> >> Just to be sure, can you reproduce the issue without cue_cut?
> >>> > Of course, i have disabled it and restarted all stations, i will
> >>> > continue
> >>> > reporting if liquidsoap processes will die again.
> >>> >
> >>> >
> >>> > Hope this information helps. Thanks for your response!
> >>> >
> >>> >
> >>> > 2014-01-29 David Baelde <david.bae...@gmail.com>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> No clue so far, so I'll ask a few questions instead.
> >>> >>
> >>> >> * Are you using the standard resolver for http, that is wget?
> >>> >>
> >>> >> * Do you have a reasonable number of alive requests? You can see
> this
> >>> >> using the "request.alive" server command. On the server you can also
> >>> >> perform "request.trace" to see what happened to a request.
> >>> >>
> >>> >> * Can you identify the too many files in "ls -l /proc/<liq pid>/fd"?
> >>> >> If they are the downloaded files, you should see temporary file
> names
> >>> >> starting with "liq". It would be useful to check if these files are
> >>> >> all the downloaded files so far or only some.
> >>> >>
> >>> >> * Just to be sure, can you reproduce the issue without cue_cut?
> >>> >>
> >>> >> Thanks,
> >>> >>
> >>> >> David
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> ------------------------------------------------------------------------------
> >>> >> WatchGuard Dimension instantly turns raw network data into
> actionable
> >>> >> security intelligence. It gives you real-time visual feedback on key
> >>> >> security issues and trends. Skip the complicated setup - simply
> import
> >>> >> a virtual appliance and go from zero to informed in seconds.
> >>> >>
> >>> >>
> >>> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> >>> >> _______________________________________________
> >>> >> Savonet-users mailing list
> >>> >> Savonet-users@lists.sourceforge.net
> >>> >> https://lists.sourceforge.net/lists/listinfo/savonet-users
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Kind regards,
> >>> > Alexander.
> >>>
> >>>
> >>>
> >>> --
> >>> David
> >>
> >>
> >>
> >>
> >> --
> >> Kind regards,
> >> Alexander.
> >>
> >>
> ------------------------------------------------------------------------------
> >> WatchGuard Dimension instantly turns raw network data into actionable
> >> security intelligence. It gives you real-time visual feedback on key
> >> security issues and trends. Skip the complicated setup - simply import
> >> a virtual appliance and go from zero to informed in seconds.
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Savonet-users mailing list
> >> Savonet-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/savonet-users
> >>
>
>
>
> --
> David
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends. Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Savonet-users mailing list
> Savonet-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/savonet-users
>
--
Kind regards,
Alexander.
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
Savonet-users mailing list
Savonet-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/savonet-users
