Hello Denis,
I was looking forward to your feedback soon.
It seems there are mainly 2 major stress points.
One, is the stream number limit per kms instance of about 200. That
should allow about 10 or 11 users in a room with both video and
microphone but this can be overcome by clustering starting with a
docker swarm and the containers can be spread over multiple servers.
Two, is the question of how many simultanious pods from users with both
video and microphone can be accommodated on the client browser for the
room. When I ran the bash script on a laptop I created 4 guests so
that was 4 instances of Chrome each with 5 pods on them. Things got
unstable after that. So I surmise that 20-25 would be the limit there.
I don't see how that limit could be passed without some design changes.
Any ideas would be welcome.
I think Maxim mentioned that Zoom draws the entire screen on their
servers and streams that to the clients.
Dennis
On Fri, 2020-11-13 at 01:23 -0600, Denis Noctor wrote:
> Hi there Maxim... I did a test with 8 computers and 2 tablets last
> night (spread across 2 WiFis)... please don’t delete the logs on the
> OM demo server (next)... I will come back to you all with some
> feedback and pics later tomorrow (if that’s okay)... however, for
> reference... I started the process in the public room #7...start time
> around 8.22pm (12th Nov) (México... 6 hrs behind) and end time
> 9.50pm... (if you want to check the logs) .... the short version is
> that 8 users experienced relatively stable performance. Will give you
> a more detailed feedback once I deal with a personal issue. All the
> best, Denis.
>
> Sent from my iPhone
> On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <[email protected]>
> wrote:
>
> > Hello All,
> > I'll try to answer in one email :)
> >
> > On Wed, 11 Nov 2020 at 20:32, dww <[email protected]> wrote:
> > > However, Denis, I think your experiment with multiple devices
> > > would be
> > >
> > > valuable as then there is only one browser tab or window with the
> > > OM
> > >
> > > room open as a guest on each device. Perhaps that will make a
> > >
> > > difference.
> >
> > yes, this would be better test (even if "fake" camera is used)
> > >
> > > Dennis
> > >
> > >
> > >
> > > On Wed, 2020-11-11 at 08:24 -0500, dww wrote:
> > >
> > > > Thanks, Denis,
> > >
> > > >
> > >
> > > > Back on Oct. 17 Maxim provided the following Bash script to be
> > > run on
> > >
> > > > the machine with a client side browser for the psuedo guest
> > > users. (
> > >
> > > > Use another machine to create the room administratively and
> > > send
> > >
> > > > invitations) This is a far simpler way to stress test the
> > > client side
> > >
> > > > browser.
> > >
> > > >
> > >
> > > > Dennis
> > >
> > > >
> > >
> > > > Hello,
> > >
> > > >
> > >
> > > > i just have tried the following script
> > >
> > > > started as `./run10.sh 5`
> > >
> > > >
> > >
> > > > everything seems to work, but my CPU was 800% busy (all cores
> > > were
> > >
> > > > 100%
> > >
> > > > busy)
> > >
> > > >
> > >
> > > > without `--use-fake-device-for-media-stream` parameter I had
> > > lots of
> > >
> > > > permission errors due to camera was "captured" by first browser
> > >
> > > > other have reported "Camera busy" error
> > >
> > > >
> > >
> > > >
> > >
> > > > _HASH_HERE_ - should be replaced with real hash (I have created
> > >
> > > > endless
> > >
> > > > invitation hash to the private conference room)
> > >
> > > >
> > >
> > > > the script
> > >
> > > > ===============================================
> > >
> > > > #!/bin/bash
> > >
> > > >
> > >
> > > > i=$1
> > >
> > > >
> > >
> > > > if [ -z "${i}" ]; then
> > >
> > > > i=30
> > >
> > > > fi
> > >
> > > > let "i += 0"
> > >
> > > >
> > >
> > > > rm -rf /tmp/delme*
> > >
> > > >
> > >
> > > > while ((i--)); do
> > >
> > > > #echo "${i}"
> > >
> > > > mkdir /tmp/delme${i}
> > >
> > > >
> > >
> > > > #local conference
> > >
> > > > chromium-browser --user-data-dir=/tmp/delme${i} --disable-
> > > infobars
> > >
> > > > --no-default-browser-check --allow-insecure-localhost
> > >
> > > > --use-fake-device-for-media-stream '
> > >
> > > >
> > > https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1'
> > >
> > > > &
> > >
> > > > done
> > >
> > > >
> > >
> > > >
> > >
> > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote:
> > >
> > > > > Hi there everyone, this seems to be the “elephant in the
> > > room”
> > >
> > > > > discussion, while there has been a HUGE amount of development
> > > and
> > >
> > > > > progress in OM since March (thank you so much @Maxim) ...
> > > there is
> > >
> > > > > the whole issue of, for example, the number of users per
> > > room...
> > >
> > > > > which seems to be about 5-6 (and maybe even to 7) when pushed
> > > to
> > >
> > > > > the
> > >
> > > > > limit... with both audio and video being broadcasted from all
> > >
> > > > > users... and, something else.. if there are simultaneous
> > >
> > > > > classes/sessions being held on the same server... will this
> > >
> > > > > restrict
> > >
> > > > > things even further? Is this an overall limitation
> >
> > Sebastian did some AWS based testingAnd, if i'm not mistaken, the
> > server with 4GB RAM was able to handle at least 3 rooms of 5
> > people(5.1.0-SNAPSHOT should behave better than 5.0.1)
> > to increase the number of rooms you can use cluster
> > > to using a
> > >
> > > > > browser
> > >
> > > > > based approach... or should we be taking approach?
> >
> > well,there is "The Limit"KMS can handle only certain amount of
> > multimedia connectionsadditionally there are other limits:-
> > bandwidth- CPU- RAM- open files (network socket is a file)
> > "The Limit" is something I'm not sure how to deal with (yet)
> > > > >
> > >
> > > > > It was my intention to test out the OM “demo servers” over
> > > the last
> > >
> > > > > 2
> > >
> > > > > weeks but will take today off and try to test 10 real device
> > >
> > > > > connections... with a combination of desktops, laptops,
> > > android
> > >
> > > > > tablets and maybe even the odd iPhone or two.
> >
> > Apple devices has issues with sound (outgoing)I'm still
> > investigating this one
> > > > >
> > >
> > > > > My million dollar question is... prior to WebRTC and
> > > Kurento... was
> > >
> > > > > it possible to have 5-10 users in a room with audio and video
> > >
> > > > > working
> > >
> > > > > seamlessly in previous versions (for example, the old “flash”
> > > setup
> > >
> > > > > (which will be redundant after Christmas... Chrome etc
> > >
> > > > > notifications)
> > >
> > > > > and if so, what has changed?
> >
> > Yes this was possibleOM_before_5 was based on Red5 media
> > serverUnfortunately it's open source version has no WebRTC support
> > > > >
> > >
> > > > > If there is anyone out there that has no problem with user
> > > numbers
> > >
> > > > > (using audio and vid)... exceeding a body of 7-10+, please
> > > let us
> > >
> > > > > know.
> > >
> > > > >
> > >
> > > > > In the meantime, I’ll give you my feedback on my tests.
> > >
> > > > >
> > >
> > > > > I really appreciate everything that has been done to date.
> > >
> > > > >
> > >
> > > > > Thanks.
> > >
> > > > >
> > >
> > > > > Sent from my iPhone
> > >
> > > > >
> > >
> > > > > > On Nov 9, 2020, at 4:50 PM, dww <[email protected]>
> > > wrote:
> > >
> > > > > >
> > >
> > > > > > Hello Maxim,
> > >
> > > > > >
> > >
> > > > > > A couple of weeks ago there was an email thread about the 5
> > > total
> > >
> > > > > > users
> > >
> > > > > > for one room, each user with video/microphone under the
> > >
> > > > > > Subject: "docker container clustering experiments #1".
> >
> > For whatever reason you love to start new mail threads :))))
> > > In this
> > >
> > > > > > case
> > >
> > > > > > it
> > >
> > > > > > appears the bottleneck is the CPU usage on the client
> > > machine
> > >
> > > > > > with
> > >
> > > > > > the
> > >
> > > > > > browser.
> > >
> > > > > >
> > >
> > > > > > In a response to Denis Noctor on a similar thread you
> > > mentioned
> > >
> > > > > > to
> > >
> > > > > > try
> > >
> > > > > > the following:
> > >
> > > > > >
> > >
> > > > > > "please check allowed amount of opened files for the user
> > > who
> > >
> > > > > > starts
> > >
> > > > > > OM/KMS/TURN
> > >
> > > > > > increasing it might help"
> > >
> > > > > >
> > >
> > > > > > Might this help with the issue we discussed? Where
> > > approximately
> > >
> > > > > > do
> > >
> > > > > > I
> > >
> > > > > > set the allowed amount of opened files?
> >
> > KMS seems to drop connections when there is not enough
> > files(network socket is a file)you can check the limit for current
> > user using `ulimit -n` (`ulimit -a` to see all limits)
> > to check limit for `nobody` user `su nobody --shell /bin/bash --
> > command "ulimit -n"`
> > to increase the limit i'm changing `/etc/security/limits.conf` file
> > https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35
> >
> > (seems to work at Ubuntu)
> >
> > > > > >
> > >
> > > > > > Also are there any other things that can be tried to
> > > improve this
> > >
> > > > > > scalability? Are there areas in the code that can be
> > > examined to
> > >
> > > > > > investigate how to improve this?
> >
> > KMS cluster would be ultimate solution, I guess
> > > > > >
> > >
> > > > > > Thanks,
> > >
> > > > > > Dennis
> > >
> > > > > >
> > >
> > > > > >
> > >
> > >
> > >
> >
> >
