I apologize for accidently sending this twice.
On Tue, 2020-11-17 at 08:40 -0500, dww wrote:
> Hello Denis,
> I was looking forward to your feedback soon.
> It seems there are mainly 2 major stress points.
> One, is the stream number limit per kms instance of about 200. That
> should allow about 10 or 11 users in a room with both video and
> microphone but this can be overcome by clustering starting with a
> docker swarm and the containers can be spread over multiple servers.
> Two, is the question of how many simultanious pods from users with
> both video and microphone can be accommodated on the client browser
> for the room. When I ran the bash script on a laptop I created 4
> guests so that was 4 instances of Chrome each with 5 pods on them.
> Things got unstable after that. So I surmise that 20-25 would be the
> limit there. I don't see how that limit could be passed without some
> design changes.
> Any ideas would be welcome.
> I think Maxim mentioned that Zoom draws the entire screen on their
> servers and streams that to the clients.
> Dennis
> On Fri, 2020-11-13 at 01:23 -0600, Denis Noctor wrote:
> > Hi there Maxim... I did a test with 8 computers and 2 tablets last
> > night (spread across 2 WiFis)... please don’t delete the logs on
> > the OM demo server (next)... I will come back to you all with some
> > feedback and pics later tomorrow (if that’s okay)... however, for
> > reference... I started the process in the public room #7...start
> > time around 8.22pm (12th Nov) (México... 6 hrs behind) and end time
> > 9.50pm... (if you want to check the logs) .... the short version is
> > that 8 users experienced relatively stable performance. Will give
> > you a more detailed feedback once I deal with a personal issue. All
> > the best, Denis.
> >
> > Sent from my iPhone
> > On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <[email protected]
> > > wrote:
> >
> > > Hello All,
> > > I'll try to answer in one email :)
> > >
> > > On Wed, 11 Nov 2020 at 20:32, dww <[email protected]> wrote:
> > > > However, Denis, I think your experiment with multiple devices
> > > > would be
> > > >
> > > > valuable as then there is only one browser tab or window with
> > > > the OM
> > > >
> > > > room open as a guest on each device. Perhaps that will make a
> > > >
> > > > difference.
> > >
> > > yes, this would be better test (even if "fake" camera is used)
> > > >
> > > > Dennis
> > > >
> > > >
> > > >
> > > > On Wed, 2020-11-11 at 08:24 -0500, dww wrote:
> > > >
> > > > > Thanks, Denis,
> > > >
> > > > >
> > > >
> > > > > Back on Oct. 17 Maxim provided the following Bash script to
> > > > be run on
> > > >
> > > > > the machine with a client side browser for the psuedo guest
> > > > users. (
> > > >
> > > > > Use another machine to create the room administratively and
> > > > send
> > > >
> > > > > invitations) This is a far simpler way to stress test the
> > > > client side
> > > >
> > > > > browser.
> > > >
> > > > >
> > > >
> > > > > Dennis
> > > >
> > > > >
> > > >
> > > > > Hello,
> > > >
> > > > >
> > > >
> > > > > i just have tried the following script
> > > >
> > > > > started as `./run10.sh 5`
> > > >
> > > > >
> > > >
> > > > > everything seems to work, but my CPU was 800% busy (all cores
> > > > were
> > > >
> > > > > 100%
> > > >
> > > > > busy)
> > > >
> > > > >
> > > >
> > > > > without `--use-fake-device-for-media-stream` parameter I had
> > > > lots of
> > > >
> > > > > permission errors due to camera was "captured" by first
> > > > browser
> > > >
> > > > > other have reported "Camera busy" error
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > _HASH_HERE_ - should be replaced with real hash (I have
> > > > created
> > > >
> > > > > endless
> > > >
> > > > > invitation hash to the private conference room)
> > > >
> > > > >
> > > >
> > > > > the script
> > > >
> > > > > ===============================================
> > > >
> > > > > #!/bin/bash
> > > >
> > > > >
> > > >
> > > > > i=$1
> > > >
> > > > >
> > > >
> > > > > if [ -z "${i}" ]; then
> > > >
> > > > > i=30
> > > >
> > > > > fi
> > > >
> > > > > let "i += 0"
> > > >
> > > > >
> > > >
> > > > > rm -rf /tmp/delme*
> > > >
> > > > >
> > > >
> > > > > while ((i--)); do
> > > >
> > > > > #echo "${i}"
> > > >
> > > > > mkdir /tmp/delme${i}
> > > >
> > > > >
> > > >
> > > > > #local conference
> > > >
> > > > > chromium-browser --user-data-dir=/tmp/delme${i} --disable-
> > > > infobars
> > > >
> > > > > --no-default-browser-check --allow-insecure-localhost
> > > >
> > > > > --use-fake-device-for-media-stream '
> > > >
> > > > >
> > > > https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1'
> > > >
> > > > > &
> > > >
> > > > > done
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote:
> > > >
> > > > > > Hi there everyone, this seems to be the “elephant in the
> > > > room”
> > > >
> > > > > > discussion, while there has been a HUGE amount of
> > > > development and
> > > >
> > > > > > progress in OM since March (thank you so much @Maxim) ...
> > > > there is
> > > >
> > > > > > the whole issue of, for example, the number of users per
> > > > room...
> > > >
> > > > > > which seems to be about 5-6 (and maybe even to 7) when
> > > > pushed to
> > > >
> > > > > > the
> > > >
> > > > > > limit... with both audio and video being broadcasted from
> > > > all
> > > >
> > > > > > users... and, something else.. if there are simultaneous
> > > >
> > > > > > classes/sessions being held on the same server... will this
> > > >
> > > > > > restrict
> > > >
> > > > > > things even further? Is this an overall limitation
> > >
> > > Sebastian did some AWS based testingAnd, if i'm not mistaken, the
> > > server with 4GB RAM was able to handle at least 3 rooms of 5
> > > people(5.1.0-SNAPSHOT should behave better than 5.0.1)
> > > to increase the number of rooms you can use cluster
> > > > to using a
> > > >
> > > > > > browser
> > > >
> > > > > > based approach... or should we be taking approach?
> > >
> > > well,there is "The Limit"KMS can handle only certain amount of
> > > multimedia connectionsadditionally there are other limits:-
> > > bandwidth- CPU- RAM- open files (network socket is a file)
> > > "The Limit" is something I'm not sure how to deal with (yet)
> > > > > >
> > > >
> > > > > > It was my intention to test out the OM “demo servers” over
> > > > the last
> > > >
> > > > > > 2
> > > >
> > > > > > weeks but will take today off and try to test 10 real
> > > > device
> > > >
> > > > > > connections... with a combination of desktops, laptops,
> > > > android
> > > >
> > > > > > tablets and maybe even the odd iPhone or two.
> > >
> > > Apple devices has issues with sound (outgoing)I'm still
> > > investigating this one
> > > > > >
> > > >
> > > > > > My million dollar question is... prior to WebRTC and
> > > > Kurento... was
> > > >
> > > > > > it possible to have 5-10 users in a room with audio and
> > > > video
> > > >
> > > > > > working
> > > >
> > > > > > seamlessly in previous versions (for example, the old
> > > > “flash” setup
> > > >
> > > > > > (which will be redundant after Christmas... Chrome etc
> > > >
> > > > > > notifications)
> > > >
> > > > > > and if so, what has changed?
> > >
> > > Yes this was possibleOM_before_5 was based on Red5 media
> > > serverUnfortunately it's open source version has no WebRTC
> > > support
> > > > > >
> > > >
> > > > > > If there is anyone out there that has no problem with user
> > > > numbers
> > > >
> > > > > > (using audio and vid)... exceeding a body of 7-10+, please
> > > > let us
> > > >
> > > > > > know.
> > > >
> > > > > >
> > > >
> > > > > > In the meantime, I’ll give you my feedback on my tests.
> > > >
> > > > > >
> > > >
> > > > > > I really appreciate everything that has been done to date.
> > > >
> > > > > >
> > > >
> > > > > > Thanks.
> > > >
> > > > > >
> > > >
> > > > > > Sent from my iPhone
> > > >
> > > > > >
> > > >
> > > > > > > On Nov 9, 2020, at 4:50 PM, dww <[email protected]>
> > > > wrote:
> > > >
> > > > > > >
> > > >
> > > > > > > Hello Maxim,
> > > >
> > > > > > >
> > > >
> > > > > > > A couple of weeks ago there was an email thread about the
> > > > 5 total
> > > >
> > > > > > > users
> > > >
> > > > > > > for one room, each user with video/microphone under the
> > > >
> > > > > > > Subject: "docker container clustering experiments #1".
> > >
> > > For whatever reason you love to start new mail threads :))))
> > > > In this
> > > >
> > > > > > > case
> > > >
> > > > > > > it
> > > >
> > > > > > > appears the bottleneck is the CPU usage on the client
> > > > machine
> > > >
> > > > > > > with
> > > >
> > > > > > > the
> > > >
> > > > > > > browser.
> > > >
> > > > > > >
> > > >
> > > > > > > In a response to Denis Noctor on a similar thread you
> > > > mentioned
> > > >
> > > > > > > to
> > > >
> > > > > > > try
> > > >
> > > > > > > the following:
> > > >
> > > > > > >
> > > >
> > > > > > > "please check allowed amount of opened files for the user
> > > > who
> > > >
> > > > > > > starts
> > > >
> > > > > > > OM/KMS/TURN
> > > >
> > > > > > > increasing it might help"
> > > >
> > > > > > >
> > > >
> > > > > > > Might this help with the issue we discussed? Where
> > > > approximately
> > > >
> > > > > > > do
> > > >
> > > > > > > I
> > > >
> > > > > > > set the allowed amount of opened files?
> > >
> > > KMS seems to drop connections when there is not enough
> > > files(network socket is a file)you can check the limit for
> > > current user using `ulimit -n` (`ulimit -a` to see all limits)
> > > to check limit for `nobody` user `su nobody --shell /bin/bash --
> > > command "ulimit -n"`
> > > to increase the limit i'm changing `/etc/security/limits.conf`
> > > file
> > > https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35
> > >
> > > (seems to work at Ubuntu)
> > >
> > > > > > >
> > > >
> > > > > > > Also are there any other things that can be tried to
> > > > improve this
> > > >
> > > > > > > scalability? Are there areas in the code that can be
> > > > examined to
> > > >
> > > > > > > investigate how to improve this?
> > >
> > > KMS cluster would be ultimate solution, I guess
> > > > > > >
> > > >
> > > > > > > Thanks,
> > > >
> > > > > > > Dennis
> > > >
> > > > > > >
> > > >
> > > > > > >
> > > >
> > > >
> > > >
> > >
> > >
