Hi there, /this might be because BBB uses 3+ KMS servers clustered/ /(I was unable to find direct link regarding it ...)/
as far as I know, they use freeswitch for audio and kms just for the video streams... BR Peter Am 23.11.20 um 10:53 schrieb Denis Noctor: > Thanks a lot Maxim. I am happy to help in anyway for future tests. > > Sent from my iPhone > > On Nov 23, 2020, at 3:43 AM, Maxim Solodovnik <[email protected] > <mailto:[email protected]>> wrote: > >> Thanks for very detailed report Denis >> >> couple of thoughts inline: >> >> On Mon, 23 Nov 2020 at 13:59, Denis Noctor <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi there everyone, >> >> My sincerest apologies for only getting back to you now. As I had >> mentioned in my previous email that I was going through a >> personal issue and it took longer than I had anticipated to get >> back in touch. >> >> However, as mentioned before, I have been keeping up to date by >> reading all the emails in the forum over the last few weeks... >> and some direct emails also. >> >> I also apologize for the length of the email... so maybe you >> should grab a coffee. >> >> As mentioned in my previous email, I set up 10 devices to connect >> to Room 7 on the OM demo server - all of which where running the >> latest version of Chrome. 3 machines running Windows 10, 1 >> running Windows 8, 4 running Windows 7, 2 Amazon Fires (set up to >> run Chrome) with varying degrees of ram (2, 4, 8 and 12 gigs) >> >> *The results are as follows:* >> After logging in around 8.22pm (Mexico time), 12th November >> (OpenMeetings - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build >> date: 2020-11-09T14:57:23Z , I gradually added other devices to >> the room. I got to 8. There was a little but of a time lapse... >> in the sense that I would move from one computer to another... >> and could still see myself in one feed after I had move to >> another. It is important to note that 2 of the computers (older >> HP's) have a slight webcam issue... (I think there is a fauly >> cable... sometimes works sometimes doesn't - but audio/mic was >> working fine). >> >> When I added 2 the last two devices, things started to break >> down. The audio quality was clearly reduced... there was a lot of >> crackling sounds... and some of the users video pods disappeared >> from some of the devices´screens.... or "empty" video pods filled >> some screens... on some devices.... but were viewable on others. >> Some users appeared to be disconnected, though they could >> continue to view the whiteboards... but had their audio and video >> disconnected (icons in orange)... when they tried to reconnect... >> they couldn't... they clicked on the audio / vid icons but with >> no effect... refreshing the screen sometimes seemed to correct >> this. >> >> While 8 users seemed to be able to connect okay... there was a >> little bit of a time delay. As you can understand, I don't have >> headphones and microphones for each and every computer... so I >> spaced them around my house... when I talked... I could hear my >> voice being repeated... (I am not referring to echo feedback).... >> there was a slight time delay by a couple of seconds on some of >> the devices... moving from device to device. However, with just 5 >> users in a room, this was not really an issue. >> >> From time to time users experienced other users being >> disconnected or whereby they could see the "empty video pod" with >> the green border flashing on and off as someone spoke.... but >> again no audio or video being received.... but it was possible to >> see the same users on other devices. >> >> Users would try to "refresh" the page... again only having access >> to 4 - 5 users on the page.... and not necessarily seeing the >> moderator. I finished testing around 9.50pm. >> >> *Some additional observations:* >> >> Based on some of the emails over the last few weeks. It appears >> to be that one OM instance can only deal with 3 simultaneous >> rooms with 5 users approx in each room (using audio and video)... >> and based on the above maybe a little more, but at a stretch. >> This appears to boil down to limitations due to number Kurento / >> WebRTC connections.... some of you have mention somewhere in the >> range of 200 - 300 connections. >> >> As a result I took a look at a few sites regarding BigBlueButton >> (BBB), as it also uses Kurento and WebRTC to get a general idea >> as to how many users can be in a room (with camera and audio). >> However, a lot of digging had to be done as many of the numbers >> that are used are about how many participants can be in a room >> (without cam and mic) with a moderator (using cam and mic). Now I >> apologize for bringing up BBB in conversation, as I am not >> endorsing the platform....reminding me of Harry Potter, >> (Voldemort) "He who shall not be named" :) >> >> However, it might be worth investigating for ideas on how to >> increase the number of cams / mics in an OM room. >> >> You can view this information here: >> >> https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- >> >> Similarly, if you look at the following link it suggests that the >> more number of concurrent users... the number of rooms will be >> less (but again not taking into consideration cams and mics) : >> >> https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support >> >> However, regarding the limited number of users (with cam and >> mic), BBB seems to have got arround this by having a window of 5 >> cams that can be scrolled left or right. It appears moderators >> can still view up to 25 cams etc. >> >> Take a look at this: >> >> https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates >> >> >> this might be because BBB uses 3+ KMS servers clustered >> (I was unable to find direct link regarding it ...) >> >> I guess we might contact BBB devs and ask for the help in KMS >> configuration/clustering >> >> >> >> _________________________ >> >> *OM Demo Next Server Specs:* >> >> @Maxim, is it possible to share the specs of the demo server, >> ram, cpu, etc, so that we can get a general idea as to what to >> benchmark against. >> >> >> I did share the specs before >> here they are >> >> dedicated server with >> CPU: 8x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz >> RAM: 24GB >> >> It hosting >> om-demo: 8GB for java >> om-next: 4GB for java >> >> These instances are using >> - same KMS >> - same Coturn >> - Separate databases at same MySql >> >> I can increase java memory for demo-next if you are planning to do >> tests one more time >> >> >> >> At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). >> However, I am under the impression that even if I upgrade my AWS >> server... I am not really going to see any major improvements >> given the fact that I am still limited to the number of users >> (with cam and mic) per room. >> >> Before this whole covid situation, my school had 10 classrooms, >> class size average 10. I would like to be able to have a similar >> virtual set up... but based on all the info above I would need 3 >> - 4 instances (using clustering, which I have never done before) >> - but will still have a problem having 11 users (10 students / 1 >> moderator) in a room. >> >> And while I know the following question (as an alternative) might >> be considered ridiculous - can a multiple number of OM >> installations (with multiple KMS etc) be conducted in one server >> - which more ram, cpu power etc? >> >> It would be great if anyone out there has a successful clustering >> model that they could share - even to test across 2 instances. At >> present, I am using Ubuntu 18.04 on AWS as described above. >> >> Either way it seems the main obstacle at the moment seems to be >> how Kurento and WebRTC can be set up to overcome these limitations >> >> Apologies once again for the length of this email and for taking >> so long to get back in touch. >> >> (I've added a few screenshots regarding my test below (one >> computer's time is 2 hours behind for some strange reason) :)) >> >> All the best, >> >> Denis. >> >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> Virus-free. www.avg.com >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> >> >> >> On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik >> <[email protected] <mailto:[email protected]>> wrote: >> >> >> >> On Fri, 13 Nov 2020 at 14:24, Denis Noctor >> <[email protected] <mailto:[email protected]>> wrote: >> >> Hi there Maxim... I did a test with 8 computers and 2 >> tablets last night (spread across 2 WiFis)... please >> don’t delete the logs on the OM demo server (next)... >> >> >> I just backup the logs >> >> >> I will come back to you all with some feedback and pics >> later tomorrow (if that’s okay)... however, for >> reference... I started the process in the public room >> #7...start time around 8.22pm (12th Nov) (México... 6 hrs >> behind) and end time 9.50pm... (if you want to check the >> logs) .... the short version is that 8 users experienced >> relatively stable performance. >> >> >> Looking forward to hear the full version :)) >> >> >> Will give you a more detailed feedback once I deal with a >> personal issue. All the best, Denis. >> >> Sent from my iPhone >> >> On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik >> <[email protected] <mailto:[email protected]>> wrote: >> >>> Hello All, >>> >>> I'll try to answer in one email :) >>> >>> On Wed, 11 Nov 2020 at 20:32, dww <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> However, Denis, I think your experiment with >>> multiple devices would be >>> valuable as then there is only one browser tab or >>> window with the OM >>> room open as a guest on each device. Perhaps that >>> will make a >>> difference. >>> >>> >>> yes, this would be better test (even if "fake" camera is >>> used) >>> >>> >>> >>> Dennis >>> >>> On Wed, 2020-11-11 at 08:24 -0500, dww wrote: >>> > Thanks, Denis, >>> > >>> > Back on Oct. 17 Maxim provided the following Bash >>> script to be run on >>> > the machine with a client side browser for the >>> psuedo guest users. ( >>> > Use another machine to create the room >>> administratively and send >>> > invitations) This is a far simpler way to stress >>> test the client side >>> > browser. >>> > >>> > Dennis >>> > >>> > Hello, >>> > >>> > i just have tried the following script >>> > started as `./run10.sh 5` >>> > >>> > everything seems to work, but my CPU was 800% busy >>> (all cores were >>> > 100% >>> > busy) >>> > >>> > without `--use-fake-device-for-media-stream` >>> parameter I had lots of >>> > permission errors due to camera was "captured" by >>> first browser >>> > other have reported "Camera busy" error >>> > >>> > >>> > _HASH_HERE_ - should be replaced with real hash (I >>> have created >>> > endless >>> > invitation hash to the private conference room) >>> > >>> > the script >>> > =============================================== >>> > #!/bin/bash >>> > >>> > i=$1 >>> > >>> > if [ -z "${i}" ]; then >>> > i=30 >>> > fi >>> > let "i += 0" >>> > >>> > rm -rf /tmp/delme* >>> > >>> > while ((i--)); do >>> > #echo "${i}" >>> > mkdir /tmp/delme${i} >>> > >>> > #local conference >>> > chromium-browser --user-data-dir=/tmp/delme${i} >>> --disable-infobars >>> > --no-default-browser-check --allow-insecure-localhost >>> > --use-fake-device-for-media-stream ' >>> > >>> >>> https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1' >>> > & >>> > done >>> > >>> > >>> > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: >>> > > Hi there everyone, this seems to be the >>> “elephant in the room” >>> > > discussion, while there has been a HUGE amount >>> of development and >>> > > progress in OM since March (thank you so much >>> @Maxim) ... there is >>> > > the whole issue of, for example, the number of >>> users per room... >>> > > which seems to be about 5-6 (and maybe even to >>> 7) when pushed to >>> > > the >>> > > limit... with both audio and video being >>> broadcasted from all >>> > > users... and, something else.. if there are >>> simultaneous >>> > > classes/sessions being held on the same >>> server... will this >>> > > restrict >>> > > things even further? Is this an overall limitation >>> >>> >>> Sebastian did some AWS based testing >>> And, if i'm not mistaken, the server with 4GB RAM was >>> able to handle at least 3 rooms of 5 people >>> (5.1.0-SNAPSHOT should behave better than 5.0.1) >>> >>> to increase the number of rooms you can use cluster >>> >>> >>> to using a >>> > > browser >>> > > based approach... or should we be taking approach? >>> >>> >>> well, >>> there is "The Limit" >>> KMS can handle only certain amount of multimedia connections >>> additionally there are other limits: >>> - bandwidth >>> - CPU >>> - RAM >>> - open files (network socket is a file) >>> >>> "The Limit" is something I'm not sure how to deal with (yet) >>> >>> >>> > > >>> > > It was my intention to test out the OM “demo >>> servers” over the last >>> > > 2 >>> > > weeks but will take today off and try to test 10 >>> real device >>> > > connections... with a combination of desktops, >>> laptops, android >>> > > tablets and maybe even the odd iPhone or two. >>> >>> >>> Apple devices has issues with sound (outgoing) >>> I'm still investigating this one >>> >>> >>> > > >>> > > My million dollar question is... prior to WebRTC >>> and Kurento... was >>> > > it possible to have 5-10 users in a room with >>> audio and video >>> > > working >>> > > seamlessly in previous versions (for example, >>> the old “flash” setup >>> > > (which will be redundant after Christmas... >>> Chrome etc >>> > > notifications) >>> > > and if so, what has changed? >>> >>> >>> Yes this was possible >>> OM_before_5 was based on Red5 media server >>> Unfortunately it's open source version has no WebRTC support >>> >>> >>> > > >>> > > If there is anyone out there that has no problem >>> with user numbers >>> > > (using audio and vid)... exceeding a body of >>> 7-10+, please let us >>> > > know. >>> > > >>> > > In the meantime, I’ll give you my feedback on my >>> tests. >>> > > >>> > > I really appreciate everything that has been >>> done to date. >>> > > >>> > > Thanks. >>> > > >>> > > Sent from my iPhone >>> > > >>> > > > On Nov 9, 2020, at 4:50 PM, dww >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> > > > >>> > > > Hello Maxim, >>> > > > >>> > > > A couple of weeks ago there was an email >>> thread about the 5 total >>> > > > users >>> > > > for one room, each user with video/microphone >>> under the >>> > > > Subject: "docker container clustering >>> experiments #1". >>> >>> >>> For whatever reason you love to start new mail threads :)))) >>> >>> >>> In this >>> > > > case >>> > > > it >>> > > > appears the bottleneck is the CPU usage on the >>> client machine >>> > > > with >>> > > > the >>> > > > browser. >>> > > > >>> > > > In a response to Denis Noctor on a similar >>> thread you mentioned >>> > > > to >>> > > > try >>> > > > the following: >>> > > > >>> > > > "please check allowed amount of opened files >>> for the user who >>> > > > starts >>> > > > OM/KMS/TURN >>> > > > increasing it might help" >>> > > > >>> > > > Might this help with the issue we discussed? >>> Where approximately >>> > > > do >>> > > > I >>> > > > set the allowed amount of opened files? >>> >>> >>> KMS seems to drop connections when there is not enough files >>> (network socket is a file) >>> you can check the limit for current user using `ulimit >>> -n` (`ulimit -a` to see all limits) >>> >>> to check limit for `nobody` user `su nobody --shell >>> /bin/bash --command "ulimit -n"` >>> >>> to increase the limit i'm changing >>> `/etc/security/limits.conf` file >>> >>> https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35 >>> >>> (seems to work at Ubuntu) >>> >>> >>> >>> > > > >>> > > > Also are there any other things that can be >>> tried to improve this >>> > > > scalability? Are there areas in the code that >>> can be examined to >>> > > > investigate how to improve this? >>> >>> >>> KMS cluster would be ultimate solution, I guess >>> >>> >>> > > > >>> > > > Thanks, >>> > > > Dennis >>> > > > >>> > > > >>> >>> >>> >>> -- >>> Best regards, >>> Maxim >> >> >> >> -- >> Best regards, >> Maxim >> >> >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> Virus-free. www.avg.com >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> >> >> >> >> -- >> Best regards, >> Maxim
