Thanks for that Maxim. I came across this...
https://docs.bigbluebutton.org/2.2/customize.html (scroll down to "Run three parallel Kurento media servers"). This appears to suggest that 3 KMS (audio, video and screesharing) processes on one server. Maybe this could bring us one step further? All the best. Denis On Mon, Nov 23, 2020 at 9:31 AM Maxim Solodovnik <solomax...@gmail.com> wrote: > I just have fixed "refresh" button > So audio/video stream can be "refreshed" without page reload > both user-list and video-pod buttons are works according to my tests > > On Mon, 23 Nov 2020 at 17:08, Peter Dähn <da...@vcrp.de> wrote: > >> Hi there, >> >> *this might be because BBB uses 3+ KMS servers clustered* >> *(I was unable to find direct link regarding it ...)* >> >> as far as I know, they use freeswitch for audio and kms just for the >> video streams... >> >> BR Peter >> >> >> Am 23.11.20 um 10:53 schrieb Denis Noctor: >> >> Thanks a lot Maxim. I am happy to help in anyway for future tests. >> >> Sent from my iPhone >> >> On Nov 23, 2020, at 3:43 AM, Maxim Solodovnik <solomax...@gmail.com> >> wrote: >> >> Thanks for very detailed report Denis >> >> couple of thoughts inline: >> >> On Mon, 23 Nov 2020 at 13:59, Denis Noctor <denisnoc...@gmail.com> wrote: >> >>> Hi there everyone, >>> >>> My sincerest apologies for only getting back to you now. As I had >>> mentioned in my previous email that I was going through a personal issue >>> and it took longer than I had anticipated to get back in touch. >>> >>> However, as mentioned before, I have been keeping up to date by reading >>> all the emails in the forum over the last few weeks... and some direct >>> emails also. >>> >>> I also apologize for the length of the email... so maybe you should grab >>> a coffee. >>> >>> As mentioned in my previous email, I set up 10 devices to connect to >>> Room 7 on the OM demo server - all of which where running the latest >>> version of Chrome. 3 machines running Windows 10, 1 running Windows 8, 4 >>> running Windows 7, 2 Amazon Fires (set up to run Chrome) with varying >>> degrees of ram (2, 4, 8 and 12 gigs) >>> >>> *The results are as follows:* >>> After logging in around 8.22pm (Mexico time), 12th November >>> (OpenMeetings - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build date: >>> 2020-11-09T14:57:23Z , I gradually added other devices to the room. I got >>> to 8. There was a little but of a time lapse... in the sense that I would >>> move from one computer to another... and could still see myself in one feed >>> after I had move to another. It is important to note that 2 of the >>> computers (older HP's) have a slight webcam issue... (I think there is a >>> fauly cable... sometimes works sometimes doesn't - but audio/mic was >>> working fine). >>> >>> When I added 2 the last two devices, things started to break down. The >>> audio quality was clearly reduced... there was a lot of crackling sounds... >>> and some of the users video pods disappeared from some of the >>> devices´screens.... or "empty" video pods filled some screens... on some >>> devices.... but were viewable on others. Some users appeared to be >>> disconnected, though they could continue to view the whiteboards... but had >>> their audio and video disconnected (icons in orange)... when they tried to >>> reconnect... they couldn't... they clicked on the audio / vid icons but >>> with no effect... refreshing the screen sometimes seemed to correct this. >>> >>> While 8 users seemed to be able to connect okay... there was a little >>> bit of a time delay. As you can understand, I don't have headphones and >>> microphones for each and every computer... so I spaced them around my >>> house... when I talked... I could hear my voice being repeated... (I am not >>> referring to echo feedback).... there was a slight time delay by a couple >>> of seconds on some of the devices... moving from device to device. However, >>> with just 5 users in a room, this was not really an issue. >>> >>> From time to time users experienced other users being disconnected or >>> whereby they could see the "empty video pod" with the green border flashing >>> on and off as someone spoke.... but again no audio or video being >>> received.... but it was possible to see the same users on other devices. >>> >>> Users would try to "refresh" the page... again only having access to 4 - >>> 5 users on the page.... and not necessarily seeing the moderator. I >>> finished testing around 9.50pm. >>> >>> *Some additional observations:* >>> >>> Based on some of the emails over the last few weeks. It appears to be >>> that one OM instance can only deal with 3 simultaneous rooms with 5 users >>> approx in each room (using audio and video)... and based on the above maybe >>> a little more, but at a stretch. This appears to boil down to limitations >>> due to number Kurento / WebRTC connections.... some of you have mention >>> somewhere in the range of 200 - 300 connections. >>> >>> As a result I took a look at a few sites regarding BigBlueButton (BBB), >>> as it also uses Kurento and WebRTC to get a general idea as to how many >>> users can be in a room (with camera and audio). However, a lot of digging >>> had to be done as many of the numbers that are used are about how many >>> participants can be in a room (without cam and mic) with a moderator (using >>> cam and mic). Now I apologize for bringing up BBB in conversation, as I am >>> not endorsing the platform....reminding me of Harry Potter, (Voldemort) "He >>> who shall not be named" :) >>> >>> However, it might be worth investigating for ideas on how to increase >>> the number of cams / mics in an OM room. >>> >>> You can view this information here: >>> >>> https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- >>> >>> Similarly, if you look at the following link it suggests that the more >>> number of concurrent users... the number of rooms will be less (but again >>> not taking into consideration cams and mics) : >>> >>> https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support >>> >>> However, regarding the limited number of users (with cam and mic), BBB >>> seems to have got arround this by having a window of 5 cams that can be >>> scrolled left or right. It appears moderators can still view up to 25 cams >>> etc. >>> >>> Take a look at this: >>> https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates >>> >>> >> this might be because BBB uses 3+ KMS servers clustered >> (I was unable to find direct link regarding it ...) >> >> I guess we might contact BBB devs and ask for the help in KMS >> configuration/clustering >> >> >> >>> _________________________ >>> >>> *OM Demo Next Server Specs:* >>> >>> @Maxim, is it possible to share the specs of the demo server, ram, cpu, >>> etc, so that we can get a general idea as to what to benchmark against. >>> >> >> I did share the specs before >> here they are >> >> dedicated server with >> CPU: 8x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz >> RAM: 24GB >> >> It hosting >> om-demo: 8GB for java >> om-next: 4GB for java >> >> These instances are using >> - same KMS >> - same Coturn >> - Separate databases at same MySql >> >> I can increase java memory for demo-next if you are planning to do tests >> one more time >> >> >>> >>> At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). However, >>> I am under the impression that even if I upgrade my AWS server... I am not >>> really going to see any major improvements given the fact that I am still >>> limited to the number of users (with cam and mic) per room. >>> >>> Before this whole covid situation, my school had 10 classrooms, class >>> size average 10. I would like to be able to have a similar virtual set >>> up... but based on all the info above I would need 3 - 4 instances (using >>> clustering, which I have never done before) - but will still have a problem >>> having 11 users (10 students / 1 moderator) in a room. >>> >>> And while I know the following question (as an alternative) might be >>> considered ridiculous - can a multiple number of OM installations (with >>> multiple KMS etc) be conducted in one server - which more ram, cpu power >>> etc? >>> >>> It would be great if anyone out there has a successful clustering model >>> that they could share - even to test across 2 instances. At present, I am >>> using Ubuntu 18.04 on AWS as described above. >>> >>> Either way it seems the main obstacle at the moment seems to be how >>> Kurento and WebRTC can be set up to overcome these limitations >>> >>> Apologies once again for the length of this email and for taking so long >>> to get back in touch. >>> >>> (I've added a few screenshots regarding my test below (one computer's >>> time is 2 hours behind for some strange reason) :)) >>> >>> All the best, >>> >>> Denis. >>> >>> >>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>> Virus-free. >>> www.avg.com >>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>> >>> On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik <solomax...@gmail.com> >>> wrote: >>> >>>> >>>> >>>> On Fri, 13 Nov 2020 at 14:24, Denis Noctor <denisnoc...@gmail.com> >>>> wrote: >>>> >>>>> Hi there Maxim... I did a test with 8 computers and 2 tablets last >>>>> night (spread across 2 WiFis)... please don’t delete the logs on the OM >>>>> demo server (next)... >>>>> >>>> >>>> I just backup the logs >>>> >>>> >>>>> I will come back to you all with some feedback and pics later tomorrow >>>>> (if that’s okay)... however, for reference... I started the process in the >>>>> public room #7...start time around 8.22pm (12th Nov) (México... 6 hrs >>>>> behind) and end time 9.50pm... (if you want to check the logs) .... the >>>>> short version is that 8 users experienced relatively stable performance. >>>>> >>>> >>>> Looking forward to hear the full version :)) >>>> >>>> >>>>> Will give you a more detailed feedback once I deal with a personal >>>>> issue. All the best, Denis. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <solomax...@gmail.com> >>>>> wrote: >>>>> >>>>> Hello All, >>>>> >>>>> I'll try to answer in one email :) >>>>> >>>>> On Wed, 11 Nov 2020 at 20:32, dww <dwort...@mykolab.com> wrote: >>>>> >>>>>> However, Denis, I think your experiment with multiple devices would be >>>>>> valuable as then there is only one browser tab or window with the OM >>>>>> room open as a guest on each device. Perhaps that will make a >>>>>> difference. >>>>>> >>>>> >>>>> yes, this would be better test (even if "fake" camera is used) >>>>> >>>>> >>>>>> >>>>>> Dennis >>>>>> >>>>>> On Wed, 2020-11-11 at 08:24 -0500, dww wrote: >>>>>> > Thanks, Denis, >>>>>> > >>>>>> > Back on Oct. 17 Maxim provided the following Bash script to be run >>>>>> on >>>>>> > the machine with a client side browser for the psuedo guest users. ( >>>>>> > Use another machine to create the room administratively and send >>>>>> > invitations) This is a far simpler way to stress test the client >>>>>> side >>>>>> > browser. >>>>>> > >>>>>> > Dennis >>>>>> > >>>>>> > Hello, >>>>>> > >>>>>> > i just have tried the following script >>>>>> > started as `./run10.sh 5` >>>>>> > >>>>>> > everything seems to work, but my CPU was 800% busy (all cores were >>>>>> > 100% >>>>>> > busy) >>>>>> > >>>>>> > without `--use-fake-device-for-media-stream` parameter I had lots of >>>>>> > permission errors due to camera was "captured" by first browser >>>>>> > other have reported "Camera busy" error >>>>>> > >>>>>> > >>>>>> > _HASH_HERE_ - should be replaced with real hash (I have created >>>>>> > endless >>>>>> > invitation hash to the private conference room) >>>>>> > >>>>>> > the script >>>>>> > =============================================== >>>>>> > #!/bin/bash >>>>>> > >>>>>> > i=$1 >>>>>> > >>>>>> > if [ -z "${i}" ]; then >>>>>> > i=30 >>>>>> > fi >>>>>> > let "i += 0" >>>>>> > >>>>>> > rm -rf /tmp/delme* >>>>>> > >>>>>> > while ((i--)); do >>>>>> > #echo "${i}" >>>>>> > mkdir /tmp/delme${i} >>>>>> > >>>>>> > #local conference >>>>>> > chromium-browser --user-data-dir=/tmp/delme${i} --disable-infobars >>>>>> > --no-default-browser-check --allow-insecure-localhost >>>>>> > --use-fake-device-for-media-stream ' >>>>>> > >>>>>> https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1 >>>>>> ' >>>>>> > & >>>>>> > done >>>>>> > >>>>>> > >>>>>> > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: >>>>>> > > Hi there everyone, this seems to be the “elephant in the room” >>>>>> > > discussion, while there has been a HUGE amount of development and >>>>>> > > progress in OM since March (thank you so much @Maxim) ... there is >>>>>> > > the whole issue of, for example, the number of users per room... >>>>>> > > which seems to be about 5-6 (and maybe even to 7) when pushed to >>>>>> > > the >>>>>> > > limit... with both audio and video being broadcasted from all >>>>>> > > users... and, something else.. if there are simultaneous >>>>>> > > classes/sessions being held on the same server... will this >>>>>> > > restrict >>>>>> > > things even further? Is this an overall limitation >>>>> >>>>> >>>>> Sebastian did some AWS based testing >>>>> And, if i'm not mistaken, the server with 4GB RAM was able to handle >>>>> at least 3 rooms of 5 people >>>>> (5.1.0-SNAPSHOT should behave better than 5.0.1) >>>>> >>>>> to increase the number of rooms you can use cluster >>>>> >>>>> >>>>>> to using a >>>>>> > > browser >>>>>> > > based approach... or should we be taking approach? >>>>>> >>>>> >>>>> well, >>>>> there is "The Limit" >>>>> KMS can handle only certain amount of multimedia connections >>>>> additionally there are other limits: >>>>> - bandwidth >>>>> - CPU >>>>> - RAM >>>>> - open files (network socket is a file) >>>>> >>>>> "The Limit" is something I'm not sure how to deal with (yet) >>>>> >>>>> >>>>>> > > >>>>>> > > It was my intention to test out the OM “demo servers” over the >>>>>> last >>>>>> > > 2 >>>>>> > > weeks but will take today off and try to test 10 real device >>>>>> > > connections... with a combination of desktops, laptops, android >>>>>> > > tablets and maybe even the odd iPhone or two. >>>>>> >>>>> >>>>> Apple devices has issues with sound (outgoing) >>>>> I'm still investigating this one >>>>> >>>>> >>>>>> > > >>>>>> > > My million dollar question is... prior to WebRTC and Kurento... >>>>>> was >>>>>> > > it possible to have 5-10 users in a room with audio and video >>>>>> > > working >>>>>> > > seamlessly in previous versions (for example, the old “flash” >>>>>> setup >>>>>> > > (which will be redundant after Christmas... Chrome etc >>>>>> > > notifications) >>>>>> > > and if so, what has changed? >>>>>> >>>>> >>>>> Yes this was possible >>>>> OM_before_5 was based on Red5 media server >>>>> Unfortunately it's open source version has no WebRTC support >>>>> >>>>> >>>>>> > > >>>>>> > > If there is anyone out there that has no problem with user numbers >>>>>> > > (using audio and vid)... exceeding a body of 7-10+, please let us >>>>>> > > know. >>>>>> > > >>>>>> > > In the meantime, I’ll give you my feedback on my tests. >>>>>> > > >>>>>> > > I really appreciate everything that has been done to date. >>>>>> > > >>>>>> > > Thanks. >>>>>> > > >>>>>> > > Sent from my iPhone >>>>>> > > >>>>>> > > > On Nov 9, 2020, at 4:50 PM, dww <dwort...@mykolab.com> wrote: >>>>>> > > > >>>>>> > > > Hello Maxim, >>>>>> > > > >>>>>> > > > A couple of weeks ago there was an email thread about the 5 >>>>>> total >>>>>> > > > users >>>>>> > > > for one room, each user with video/microphone under the >>>>>> > > > Subject: "docker container clustering experiments #1". >>>>> >>>>> >>>>> For whatever reason you love to start new mail threads :)))) >>>>> >>>>> >>>>>> In this >>>>>> > > > case >>>>>> > > > it >>>>>> > > > appears the bottleneck is the CPU usage on the client machine >>>>>> > > > with >>>>>> > > > the >>>>>> > > > browser. >>>>>> > > > >>>>>> > > > In a response to Denis Noctor on a similar thread you mentioned >>>>>> > > > to >>>>>> > > > try >>>>>> > > > the following: >>>>>> > > > >>>>>> > > > "please check allowed amount of opened files for the user who >>>>>> > > > starts >>>>>> > > > OM/KMS/TURN >>>>>> > > > increasing it might help" >>>>>> > > > >>>>>> > > > Might this help with the issue we discussed? Where approximately >>>>>> > > > do >>>>>> > > > I >>>>>> > > > set the allowed amount of opened files? >>>>>> >>>>> >>>>> KMS seems to drop connections when there is not enough files >>>>> (network socket is a file) >>>>> you can check the limit for current user using `ulimit -n` (`ulimit >>>>> -a` to see all limits) >>>>> >>>>> to check limit for `nobody` user `su nobody --shell /bin/bash >>>>> --command "ulimit -n"` >>>>> >>>>> to increase the limit i'm changing `/etc/security/limits.conf` file >>>>> >>>>> https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35 >>>>> >>>>> (seems to work at Ubuntu) >>>>> >>>>> >>>>> >>>>>> > > > >>>>>> > > > Also are there any other things that can be tried to improve >>>>>> this >>>>>> > > > scalability? Are there areas in the code that can be examined to >>>>>> > > > investigate how to improve this? >>>>>> >>>>> >>>>> KMS cluster would be ultimate solution, I guess >>>>> >>>>> >>>>>> > > > >>>>>> > > > Thanks, >>>>>> > > > Dennis >>>>>> > > > >>>>>> > > > >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Maxim >>>>> >>>>> >>>> >>>> -- >>>> Best regards, >>>> Maxim >>>> >>> >>> >>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>> Virus-free. >>> www.avg.com >>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>> >> >> >> -- >> Best regards, >> Maxim >> >> >> > > -- > Best regards, > Maxim >
