KMS is being developed by Openvidu they do have cluster https://openvidu.io/pricing
the goal is to find open-sourced one :)) On Tue, 24 Nov 2020 at 15:53, Rohrbach, Gerald <[email protected]> wrote: > Denis, Maxim, > > > > yes in BBB ist´written that there are 3 Kurento Media Servers. > > > > But reading details, it´s not clustered or load balanced. > > Each server is dedicated for an media type. > > I do not know, how this can be established in OM. > > > > But maybe this is not that complex. However the traffic needs to be > redirected > > somehow, or for each media type a own port is needed. > > > > Probably a clustering is much more complex. NGINX is good for balancing > maybe, > > I think also for udp traffic. But clustering and balancing can only work > for one room, so I guess > > It won´t help if one room has much users. The setup will be very complex, > error searching really difficult. > > > > > > Gerald > > > > > > > > > > > > > > *Von:* Denis Noctor [mailto:[email protected]] > *Gesendet:* Montag, 23. November 2020 22:44 > *An:* [email protected] > *Betreff:* Re: Users per room client browser scalability > > > > Thanks for that Maxim. > > I came across this... > > https://docs.bigbluebutton.org/2.2/customize.html > > > > (scroll down to "Run three parallel Kurento media servers"). > > This appears to suggest that 3 KMS (audio, video and screesharing) > processes on one server. Maybe this could bring us one step further? > > All the best. > > > > Denis > > > > On Mon, Nov 23, 2020 at 9:31 AM Maxim Solodovnik <[email protected]> > wrote: > > I just have fixed "refresh" button > > So audio/video stream can be "refreshed" without page reload > > both user-list and video-pod buttons are works according to my tests > > > > On Mon, 23 Nov 2020 at 17:08, Peter Dähn <[email protected]> wrote: > > Hi there, > > > > *this might be because BBB uses 3+ KMS servers clustered* > > *(I was unable to find direct link regarding it ...)* > > > as far as I know, they use freeswitch for audio and kms just for the video > streams... > > BR Peter > > Am 23.11.20 um 10:53 schrieb Denis Noctor: > > Thanks a lot Maxim. I am happy to help in anyway for future tests. > > Sent from my iPhone > > > On Nov 23, 2020, at 3:43 AM, Maxim Solodovnik <[email protected]> > wrote: > > Thanks for very detailed report Denis > > > > couple of thoughts inline: > > > > On Mon, 23 Nov 2020 at 13:59, Denis Noctor <[email protected]> wrote: > > Hi there everyone, > > My sincerest apologies for only getting back to you now. As I had > mentioned in my previous email that I was going through a personal issue > and it took longer than I had anticipated to get back in touch. > > However, as mentioned before, I have been keeping up to date by reading > all the emails in the forum over the last few weeks... and some direct > emails also. > > I also apologize for the length of the email... so maybe you should grab a > coffee. > > As mentioned in my previous email, I set up 10 devices to connect to Room > 7 on the OM demo server - all of which where running the latest version of > Chrome. 3 machines running Windows 10, 1 running Windows 8, 4 running > Windows 7, 2 Amazon Fires (set up to run Chrome) with varying degrees of > ram (2, 4, 8 and 12 gigs) > > *The results are as follows:* > After logging in around 8.22pm (Mexico time), 12th November (OpenMeetings > - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build date: 2020-11-09T14:57:23Z > , I gradually added other devices to the room. I got to 8. There was a > little but of a time lapse... in the sense that I would move from one > computer to another... and could still see myself in one feed after I had > move to another. It is important to note that 2 of the computers (older > HP's) have a slight webcam issue... (I think there is a fauly cable... > sometimes works sometimes doesn't - but audio/mic was working fine). > > When I added 2 the last two devices, things started to break down. The > audio quality was clearly reduced... there was a lot of crackling sounds... > and some of the users video pods disappeared from some of the > devices´screens.... or "empty" video pods filled some screens... on some > devices.... but were viewable on others. Some users appeared to be > disconnected, though they could continue to view the whiteboards... but had > their audio and video disconnected (icons in orange)... when they tried to > reconnect... they couldn't... they clicked on the audio / vid icons but > with no effect... refreshing the screen sometimes seemed to correct this. > > While 8 users seemed to be able to connect okay... there was a little bit > of a time delay. As you can understand, I don't have headphones and > microphones for each and every computer... so I spaced them around my > house... when I talked... I could hear my voice being repeated... (I am not > referring to echo feedback).... there was a slight time delay by a couple > of seconds on some of the devices... moving from device to device. However, > with just 5 users in a room, this was not really an issue. > > From time to time users experienced other users being disconnected or > whereby they could see the "empty video pod" with the green border flashing > on and off as someone spoke.... but again no audio or video being > received.... but it was possible to see the same users on other devices. > > Users would try to "refresh" the page... again only having access to 4 - 5 > users on the page.... and not necessarily seeing the moderator. I finished > testing around 9.50pm. > > *Some additional observations:* > > Based on some of the emails over the last few weeks. It appears to be that > one OM instance can only deal with 3 simultaneous rooms with 5 users approx > in each room (using audio and video)... and based on the above maybe a > little more, but at a stretch. This appears to boil down to limitations due > to number Kurento / WebRTC connections.... some of you have mention > somewhere in the range of 200 - 300 connections. > > As a result I took a look at a few sites regarding BigBlueButton (BBB), as > it also uses Kurento and WebRTC to get a general idea as to how many users > can be in a room (with camera and audio). However, a lot of digging had to > be done as many of the numbers that are used are about how many > participants can be in a room (without cam and mic) with a moderator (using > cam and mic). Now I apologize for bringing up BBB in conversation, as I am > not endorsing the platform....reminding me of Harry Potter, (Voldemort) "He > who shall not be named" :) > > However, it might be worth investigating for ideas on how to increase the > number of cams / mics in an OM room. > > You can view this information here: > > https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- > > Similarly, if you look at the following link it suggests that the more > number of concurrent users... the number of rooms will be less (but again > not taking into consideration cams and mics) : > > https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support > > However, regarding the limited number of users (with cam and mic), BBB > seems to have got arround this by having a window of 5 cams that can be > scrolled left or right. It appears moderators can still view up to 25 cams > etc. > > Take a look at this: > https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates > > > > this might be because BBB uses 3+ KMS servers clustered > > (I was unable to find direct link regarding it ...) > > > > I guess we might contact BBB devs and ask for the help in KMS > configuration/clustering > > > > > > _________________________ > > *OM Demo Next Server Specs:* > > @Maxim, is it possible to share the specs of the demo server, ram, cpu, > etc, so that we can get a general idea as to what to benchmark against. > > > > I did share the specs before > > here they are > > > > dedicated server with > > CPU: 8x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz > > RAM: 24GB > > > > It hosting > > om-demo: 8GB for java > > om-next: 4GB for java > > > > These instances are using > > - same KMS > > - same Coturn > > - Separate databases at same MySql > > > > I can increase java memory for demo-next if you are planning to do tests > one more time > > > > > At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). However, I > am under the impression that even if I upgrade my AWS server... I am not > really going to see any major improvements given the fact that I am still > limited to the number of users (with cam and mic) per room. > > Before this whole covid situation, my school had 10 classrooms, class size > average 10. I would like to be able to have a similar virtual set up... but > based on all the info above I would need 3 - 4 instances (using clustering, > which I have never done before) - but will still have a problem having 11 > users (10 students / 1 moderator) in a room. > > And while I know the following question (as an alternative) might be > considered ridiculous - can a multiple number of OM installations (with > multiple KMS etc) be conducted in one server - which more ram, cpu power > etc? > > It would be great if anyone out there has a successful clustering model > that they could share - even to test across 2 instances. At present, I am > using Ubuntu 18.04 on AWS as described above. > > Either way it seems the main obstacle at the moment seems to be how > Kurento and WebRTC can be set up to overcome these limitations > > Apologies once again for the length of this email and for taking so long > to get back in touch. > > (I've added a few screenshots regarding my test below (one computer's time > is 2 hours behind for some strange reason) :)) > > All the best, > > Denis. > > > > Virus-free. *www.avg.com* > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik <*[email protected]*> > wrote: > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > On Fri, 13 Nov 2020 at 14:24, Denis Noctor <*[email protected]*> > wrote: > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Hi there Maxim... I did a test with 8 computers and 2 tablets last night > (spread across 2 WiFis)... please don’t delete the logs on the OM demo > server (next)... > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > I just backup the logs > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > I will come back to you all with some feedback and pics later tomorrow (if > that’s okay)... however, for reference... I started the process in the > public room #7...start time around 8.22pm (12th Nov) (México... 6 hrs > behind) and end time 9.50pm... (if you want to check the logs) .... the > short version is that 8 users experienced relatively stable performance. > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Looking forward to hear the full version :)) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Will give you a more detailed feedback once I deal with a personal issue. > All the best, Denis. > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Sent from my iPhone > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <*[email protected]*> > wrote: > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Hello All, > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > I'll try to answer in one email :) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > On Wed, 11 Nov 2020 at 20:32, dww <*[email protected]*> wrote: > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > However, Denis, I think your experiment with multiple devices would be > valuable as then there is only one browser tab or window with the OM > room open as a guest on each device. Perhaps that will make a > difference. > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > yes, this would be better test (even if "fake" camera is used) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > Dennis > > On Wed, 2020-11-11 at 08:24 -0500, dww wrote: > > Thanks, Denis, > > > > Back on Oct. 17 Maxim provided the following Bash script to be run on > > the machine with a client side browser for the psuedo guest users. ( > > Use another machine to create the room administratively and send > > invitations) This is a far simpler way to stress test the client side > > browser. > > > > Dennis > > > > Hello, > > > > i just have tried the following script > > started as `./run10.sh 5` > > > > everything seems to work, but my CPU was 800% busy (all cores were > > 100% > > busy) > > > > without `--use-fake-device-for-media-stream` parameter I had lots of > > permission errors due to camera was "captured" by first browser > > other have reported "Camera busy" error > > > > > > _HASH_HERE_ - should be replaced with real hash (I have created > > endless > > invitation hash to the private conference room) > > > > the script > > =============================================== > > #!/bin/bash > > > > i=$1 > > > > if [ -z "${i}" ]; then > > i=30 > > fi > > let "i += 0" > > > > rm -rf /tmp/delme* > > > > while ((i--)); do > > #echo "${i}" > > mkdir /tmp/delme${i} > > > > #local conference > > chromium-browser --user-data-dir=/tmp/delme${i} --disable-infobars > > --no-default-browser-check --allow-insecure-localhost > > --use-fake-device-for-media-stream ' > > > *https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1* > ' > > & > > done > > > > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: > > > Hi there everyone, this seems to be the “elephant in the room” > > > discussion, while there has been a HUGE amount of development and > > > progress in OM since March (thank you so much @Maxim) ... there is > > > the whole issue of, for example, the number of users per room... > > > which seems to be about 5-6 (and maybe even to 7) when pushed to > > > the > > > limit... with both audio and video being broadcasted from all > > > users... and, something else.. if there are simultaneous > > > classes/sessions being held on the same server... will this > > > restrict > > > things even further? Is this an overall limitation > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Sebastian did some AWS based testing > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > And, if i'm not mistaken, the server with 4GB RAM was able to handle at > least 3 rooms of 5 people > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > (5.1.0-SNAPSHOT should behave better than 5.0.1) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > to increase the number of rooms you can use cluster > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > to using a > > > browser > > > based approach... or should we be taking approach? > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > well, > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > there is "The Limit" > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > KMS can handle only certain amount of multimedia connections > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > additionally there are other limits: > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > - bandwidth > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > - CPU > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > - RAM > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > - open files (network socket is a file) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > "The Limit" is something I'm not sure how to deal with (yet) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > > It was my intention to test out the OM “demo servers” over the last > > > 2 > > > weeks but will take today off and try to test 10 real device > > > connections... with a combination of desktops, laptops, android > > > tablets and maybe even the odd iPhone or two. > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Apple devices has issues with sound (outgoing) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > I'm still investigating this one > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > > My million dollar question is... prior to WebRTC and Kurento... was > > > it possible to have 5-10 users in a room with audio and video > > > working > > > seamlessly in previous versions (for example, the old “flash” setup > > > (which will be redundant after Christmas... Chrome etc > > > notifications) > > > and if so, what has changed? > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Yes this was possible > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > OM_before_5 was based on Red5 media server > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Unfortunately it's open source version has no WebRTC support > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > > If there is anyone out there that has no problem with user numbers > > > (using audio and vid)... exceeding a body of 7-10+, please let us > > > know. > > > > > > In the meantime, I’ll give you my feedback on my tests. > > > > > > I really appreciate everything that has been done to date. > > > > > > Thanks. > > > > > > Sent from my iPhone > > > > > > > On Nov 9, 2020, at 4:50 PM, dww <*[email protected]*> wrote: > > > > > > > > Hello Maxim, > > > > > > > > A couple of weeks ago there was an email thread about the 5 total > > > > users > > > > for one room, each user with video/microphone under the > > > > Subject: "docker container clustering experiments #1". > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > For whatever reason you love to start new mail threads :)))) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > In this > > > > case > > > > it > > > > appears the bottleneck is the CPU usage on the client machine > > > > with > > > > the > > > > browser. > > > > > > > > In a response to Denis Noctor on a similar thread you mentioned > > > > to > > > > try > > > > the following: > > > > > > > > "please check allowed amount of opened files for the user who > > > > starts > > > > OM/KMS/TURN > > > > increasing it might help" > > > > > > > > Might this help with the issue we discussed? Where approximately > > > > do > > > > I > > > > set the allowed amount of opened files? > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > KMS seems to drop connections when there is not enough files > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > (network socket is a file) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > you can check the limit for current user using `ulimit -n` (`ulimit -a` to > see all limits) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > to check limit for `nobody` user `su nobody --shell /bin/bash --command > "ulimit -n"` > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > to increase the limit i'm changing `/etc/security/limits.conf` file > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > *https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35* > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > (seems to work at Ubuntu) > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > > > > Also are there any other things that can be tried to improve this > > > > scalability? Are there areas in the code that can be examined to > > > > investigate how to improve this? > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > KMS cluster would be ultimate solution, I guess > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > > > > Thanks, > > > > Dennis > > > > > > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > -- > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Best regards, > Maxim > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > -- > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Best regards, > Maxim > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Virus-free. *www.avg.com* > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > -- > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Best regards, > Maxim > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > -- > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > Best regards, > Maxim > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > -- Best regards, Maxim
