Hi again,
I have pushed the changes to support poll() in duppy. You should be able to
test it easily if you use opam to install liquidsoap:
% git clone https://github.com/savonet/ocaml-duppy.git
% cd ocaml-duppy
% opam pin add duppy .
The implementation hasn't obviously been tested heavily but it seems to be
working nice. Would love to have more feedback on it.
Romain
2017-12-04 7:57 GMT-06:00 Romain Beauxis <[email protected]>:
> Hi Marty,
>
> Thanks for pointing this out. I'm gonna be exploring the poll() function
> instead of select() to remedy this situation in future releases.
>
> Romain
>
> 2017-11-25 21:14 GMT-06:00 Marty Gauvin <[email protected]>:
>
>> This one has been resolved. A few points.
>>
>>
>>
>> 1. Increasing the number of file descriptors above 1024 doesn’t
>> help. Select can’t use them.
>>
>> 2. Solved the problem by splitting the work into a number of
>> different scripts each running in a separate liquidsoap process. Each
>> process can run about 60 input.harbor and 30 output.icecast before things
>> start to break.
>>
>>
>>
>>
>>
>> *From:* Marty Gauvin [mailto:[email protected]]
>> *Sent:* Friday, 17 November 2017 9:17 PM
>> *To:* [email protected]
>> *Subject:* [Savonet-users] liquidsoap 1.3.3 crashs with 'exception
>> invalid argument in select()
>>
>>
>>
>> I’m running in an environment where I need to have lots of harbor
>> endpoints.
>>
>>
>>
>> I have found that I get this error when I go above a certain number
>> (about 150 or so).
>>
>>
>>
>> The script is
>>
>> # =========================================
>>
>> # SCRIPT CONSTANTS
>>
>> # =========================================
>>
>> # Load in the env variables including ICE_HOST, ICE_PASS, home dir, etc
>>
>> %include ".env"
>>
>>
>>
>> # =========================================
>>
>> # RUNTIME SETTINGS
>>
>> # http://savonet.sourceforge.net/doc-svn/in_production.html
>>
>> # =========================================
>>
>> # Harbor settings
>>
>> set("harbor.verbose",true) # make sure Harbor shows EVERYTHING
>>
>> set("harbor.bind_addr", HARBOR_IP) # bind Harbor to nominated IP
>>
>> set("harbor.reverse_dns",false) # tweak to improve network slightly
>>
>>
>>
>> # Setup logging - default is /var/log/liquidsoap/<script.log>
>>
>> set("log.file.path","/var/log/liquidsoap/<script>.log")
>>
>> set("log.file",true)
>>
>> set("log.file.append",true) # don't just overwrite
>>
>> set("log.level",LOG_LEVEL)
>>
>> set("log.stdout",true) # show in console, too
>>
>> set("log.unix_timestamps",false) # show UNIX timestamps instead
>>
>> set("init.daemon.pidfile.path","/var/run/liquidsoap")
>>
>>
>>
>> # Setup decoder order
>>
>> set("decoder.file_decoders",["MP3","OGG","AAC","MP4","META","WAV","MIDI","FLAC"])
>> # set order according to likelihood
>>
>> set("decoder.stream_decoders",["MP3","OGG","AAC","WAV","PCM/BASIC","FLAC"])
>> # set order according to likelihood
>>
>>
>>
>> # =========================================
>>
>> # OUTPUT APPLICATIONS
>>
>> # =========================================
>>
>> # Build the output control - Vorbis format by default
>>
>> #out = output.icecast(%vorbis(samplerate=44100, channels=2, quality=0),
>>
>> out = output.icecast(%mp3.vbr(samplerate=44100,quality=6),
>>
>> host=ICE_HOST,
>>
>> port=ICE_PORT,
>>
>> password=ICE_PASS,
>>
>> fallible=true,
>>
>> icy_metadata="true")
>>
>>
>>
>> # =========================================
>>
>> # INCLUDES
>>
>> # =========================================
>>
>> # Load all the custom mount point includes - these need to be loaded last
>>
>> %include "macmount.inc"
>>
>> %include "mounts.inc"
>>
>>
>>
>> The includes are:
>>
>> *.env*
>>
>> # Liquidsoap home directory
>>
>> # lshomedir = "/home/liquidsoap/liquidsoap"
>>
>>
>>
>> # Icecast host settings
>>
>> ICE_HOST = "###############" # Icecast host DNS
>>
>> #ICE_USER = "source" # icecast default user is always "source" anyway
>>
>> ICE_PASS = "#########" # default Icecast password
>>
>> ICE_PORT = 7034 # Icecast listen port (not the streaming port)
>>
>>
>>
>> # Basic script settings
>>
>> LOG_LEVEL = 5 # 4 = debug-level logs; 5 = extra
>>
>>
>>
>> # Harbor config
>>
>> HARBOR_IP = "0.0.0.0" # Harbor to listen on all IPs
>>
>>
>>
>> *macmount.inc*
>>
>> def macmount(num,mntname)
>>
>> rport = num * 2
>>
>> mport = rport + 7500
>>
>> vport = rport + 7800
>>
>> vname=mntname ^ "voice"
>>
>> harbom = audio_to_stereo(input.harbor(mntname, icy=true, port=mport,
>> password=ICE_PASS, max=20.))
>>
>> harbov = strip_blank(max_blank=5.,
>> mksafe(audio_to_stereo(input.harbor(vname,
>> icy=true, port=vport, password=ICE_PASS, max=20.))))
>>
>> output=out(mount=vname,harbov)
>>
>> radio = fallback(track_sensitive=false, [harbov, harbom,
>> single("media/silence.m4a")])
>>
>> output=out(mount=mntname,radio)
>>
>> end
>>
>>
>>
>> *mounts.inc*
>>
>> macmount(1,"2IdxhCR0SLF8ks")
>>
>> macmount(2,"osmQm9AHTPxHbs")
>>
>> # further 98 rows following the same pattern
>>
>>
>>
>> The log file produced looks like this before the crash:
>>
>>
>>
>> liquidsoap@bluejay-up-pr-01:~/liquidsoap$ tail -n 100 log1.log
>>
>> 2017/11/17 10:28:33 [source:4] Source fallback_7507 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7504:4] Activations changed:
>> static=[apr5eXRHAnjs49voice:apr5eXRHAnjs49voice, strip_blank_7504],
>> dynamic=[fallback_7507:apr5eXRHAnjs49:apr5eXRHAnjs49].
>>
>> 2017/11/17 10:28:33 [source:4] Source audio_to_stereo_7499 gets up.
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7499:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source input.harbor_7498 gets up.
>>
>> 2017/11/17 10:28:33 [input.harbor_7498:4] Content kind is
>> {audio=1+;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7536 with icy = false
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7537 with icy = true
>>
>> 2017/11/17 10:28:33 [harbor:3] Adding mountpoint '/apr5eXRHAnjs49' on
>> port 7536
>>
>> 2017/11/17 10:28:33 [input(dot)harbor_7498:4] Activations changed:
>> static=[audio_to_stereo_7499:fallback_7507:apr5eXRHAnjs49:apr5eXRHAnjs49],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7499:4] Activations changed:
>> static=[], dynamic=[fallback_7507:apr5eXRHAnjs49:apr5eXRHAnjs49].
>>
>> 2017/11/17 10:28:33 [source:4] Source single_7506 gets up.
>>
>> 2017/11/17 10:28:33 [single_7506:3] "media/silence.m4a" is static,
>> resolving once for all...
>>
>> 2017/11/17 10:28:33 [single_7506:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [single_7506:4] Activations changed: static=[],
>> dynamic=[fallback_7507:apr5eXRHAnjs49:apr5eXRHAnjs49].
>>
>> 2017/11/17 10:28:33 [fallback_7507:4] Activations changed:
>> static=[apr5eXRHAnjs49:apr5eXRHAnjs49], dynamic=[].
>>
>> 2017/11/17 10:28:33 [apr5eXRHAnjs49:4] Activations changed:
>> static=[apr5eXRHAnjs49], dynamic=[].
>>
>> 2017/11/17 10:28:33 [apr5eXRHAnjs49:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source strip_blank_7493 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7493:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source mksafe gets up.
>>
>> 2017/11/17 10:28:33 [source:4] Source audio_to_stereo_7490 gets up.
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7490:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source input.harbor_7489 gets up.
>>
>> 2017/11/17 10:28:33 [input.harbor_7489:4] Content kind is
>> {audio=1+;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7834 with icy = false
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7835 with icy = true
>>
>> 2017/11/17 10:28:33 [harbor:3] Adding mountpoint '/nxYjWRxEteOgunvoice'
>> on port 7834
>>
>> 2017/11/17 10:28:33 [input(dot)harbor_7489:4] Activations changed:
>> static=[audio_to_stereo_7490:mksafe:strip_blank_7493:strip_blank_7493],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7490:4] Activations changed:
>> static=[], dynamic=[mksafe:strip_blank_7493:strip_blank_7493].
>>
>> 2017/11/17 10:28:33 [source:4] Source safe_blank gets up.
>>
>> 2017/11/17 10:28:33 [safe_blank:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [safe_blank:4] Activations changed: static=[],
>> dynamic=[mksafe:strip_blank_7493:strip_blank_7493].
>>
>> 2017/11/17 10:28:33 [mksafe:4] Activations changed:
>> static=[strip_blank_7493:strip_blank_7493], dynamic=[].
>>
>> 2017/11/17 10:28:33 [strip_blank_7493:4] Activations changed:
>> static=[strip_blank_7493], dynamic=[].
>>
>> 2017/11/17 10:28:33 [strip_blank_7493:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source output.icecast_7494 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7493:4] Activations changed:
>> static=[nxYjWRxEteOgunvoice:nxYjWRxEteOgunvoice, strip_blank_7493],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [nxYjWRxEteOgunvoice:4] Activations changed:
>> static=[nxYjWRxEteOgunvoice], dynamic=[].
>>
>> 2017/11/17 10:28:33 [nxYjWRxEteOgunvoice:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source output.icecast_7497 gets up.
>>
>> 2017/11/17 10:28:33 [source:4] Source fallback_7496 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7493:4] Activations changed:
>> static=[nxYjWRxEteOgunvoice:nxYjWRxEteOgunvoice, strip_blank_7493],
>> dynamic=[fallback_7496:nxYjWRxEteOgun:nxYjWRxEteOgun].
>>
>> 2017/11/17 10:28:33 [source:4] Source audio_to_stereo_7488 gets up.
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7488:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source input.harbor_7487 gets up.
>>
>> 2017/11/17 10:28:33 [input.harbor_7487:4] Content kind is
>> {audio=1+;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7534 with icy = false
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7535 with icy = true
>>
>> 2017/11/17 10:28:33 [harbor:3] Adding mountpoint '/nxYjWRxEteOgun' on
>> port 7534
>>
>> 2017/11/17 10:28:33 [input(dot)harbor_7487:4] Activations changed:
>> static=[audio_to_stereo_7488:fallback_7496:nxYjWRxEteOgun:nxYjWRxEteOgun],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7488:4] Activations changed:
>> static=[], dynamic=[fallback_7496:nxYjWRxEteOgun:nxYjWRxEteOgun].
>>
>> 2017/11/17 10:28:33 [source:4] Source single_7495 gets up.
>>
>> 2017/11/17 10:28:33 [single_7495:3] "media/silence.m4a" is static,
>> resolving once for all...
>>
>> 2017/11/17 10:28:33 [single_7495:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [single_7495:4] Activations changed: static=[],
>> dynamic=[fallback_7496:nxYjWRxEteOgun:nxYjWRxEteOgun].
>>
>> 2017/11/17 10:28:33 [fallback_7496:4] Activations changed:
>> static=[nxYjWRxEteOgun:nxYjWRxEteOgun], dynamic=[].
>>
>> 2017/11/17 10:28:33 [nxYjWRxEteOgun:4] Activations changed:
>> static=[nxYjWRxEteOgun], dynamic=[].
>>
>> 2017/11/17 10:28:33 [nxYjWRxEteOgun:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source strip_blank_7482 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7482:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source mksafe gets up.
>>
>> 2017/11/17 10:28:33 [source:4] Source audio_to_stereo_7479 gets up.
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7479:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source input.harbor_7478 gets up.
>>
>> 2017/11/17 10:28:33 [input.harbor_7478:4] Content kind is
>> {audio=1+;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7832 with icy = false
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7833 with icy = true
>>
>> 2017/11/17 10:28:33 [harbor:3] Adding mountpoint '/vqyXzbCzVjHLrcvoice'
>> on port 7832
>>
>> 2017/11/17 10:28:33 [input(dot)harbor_7478:4] Activations changed:
>> static=[audio_to_stereo_7479:mksafe:strip_blank_7482:strip_blank_7482],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7479:4] Activations changed:
>> static=[], dynamic=[mksafe:strip_blank_7482:strip_blank_7482].
>>
>> 2017/11/17 10:28:33 [source:4] Source safe_blank gets up.
>>
>> 2017/11/17 10:28:33 [safe_blank:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [safe_blank:4] Activations changed: static=[],
>> dynamic=[mksafe:strip_blank_7482:strip_blank_7482].
>>
>> 2017/11/17 10:28:33 [mksafe:4] Activations changed:
>> static=[strip_blank_7482:strip_blank_7482], dynamic=[].
>>
>> 2017/11/17 10:28:33 [strip_blank_7482:4] Activations changed:
>> static=[strip_blank_7482], dynamic=[].
>>
>> 2017/11/17 10:28:33 [strip_blank_7482:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source output.icecast_7483 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7482:4] Activations changed:
>> static=[vqyXzbCzVjHLrcvoice:vqyXzbCzVjHLrcvoice, strip_blank_7482],
>> dynamic=[].
>>
>> 2017/11/17 10:28:33 [vqyXzbCzVjHLrcvoice:4] Activations changed:
>> static=[vqyXzbCzVjHLrcvoice], dynamic=[].
>>
>> 2017/11/17 10:28:33 [vqyXzbCzVjHLrcvoice:4] Enabling caching mode: active
>> source.
>>
>> 2017/11/17 10:28:33 [source:4] Source output.icecast_7486 gets up.
>>
>> 2017/11/17 10:28:33 [source:4] Source fallback_7485 gets up.
>>
>> 2017/11/17 10:28:33 [strip_blank_7482:4] Activations changed:
>> static=[vqyXzbCzVjHLrcvoice:vqyXzbCzVjHLrcvoice, strip_blank_7482],
>> dynamic=[fallback_7485:vqyXzbCzVjHLrc:vqyXzbCzVjHLrc].
>>
>> 2017/11/17 10:28:33 [source:4] Source audio_to_stereo_7477 gets up.
>>
>> 2017/11/17 10:28:33 [audio_to_stereo_7477:4] Content kind is
>> {audio=2;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [source:4] Source input.harbor_7476 gets up.
>>
>> 2017/11/17 10:28:33 [input.harbor_7476:4] Content kind is
>> {audio=1+;video=0;midi=0}.
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7532 with icy = false
>>
>> 2017/11/17 10:28:33 [harbor:4] Opening port 7533 with icy = true
>>
>> 2017/11/17 10:28:33 [harbor:3] Adding mountpoint '/vqyXzbCzVjHLrc' on
>> port 7532
>>
>> 2017/11/17 10:28:33 [threads:2] Queue generic queue #1 crashed with
>> exception Invalid argument in select()
>>
>> Raised by primitive operation at file "duppy.ml", line 181, characters
>> 20-51
>>
>> Re-raised at file "duppy.ml", line 199, characters 11-18
>>
>> Called from file "duppy.ml", line 201, characters 4-8
>>
>> Called from file "duppy.ml", line 295, characters 12-25
>>
>> Called from file "tools/tutils.ml", line 213, characters 19-55
>>
>>
>>
>> 2017/11/17 10:28:33 [threads:1] PANIC: Liquidsoap has crashed, exiting.,
>>
>> Please report at: [email protected]
>>
>>
>>
>> Is this a bug, or am I doing something wrong? I upgraded from liquidsoap
>> 1.1.1 but the error I’m getting is the same.
>>
>>
>>
>> The server is on DigitalOcean running Ubuntu 16.04.03 LTS.
>>
>>
>>
>> The open files limit in ulimit has been increased to 65536.
>>
>>
>>
>> [image: cid:[email protected]]
>>
>>
>>
>>
>>
>> *Marty Gauvin*
>>
>> *President and CEO*
>>
>>
>>
>> Office +61 8 72000800 <+61%208%207200%200800>
>>
>> Mobile +61 418843550 <+61%20418%20843%20550>
>>
>> email: [email protected]
>>
>> skype: marty.gauvin.virtualark
>>
>> web: www.virtualark.com
>>
>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Savonet-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/savonet-users
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Savonet-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/savonet-users
