Denis, Maxim, yes in BBB ist´written that there are 3 Kurento Media Servers.
But reading details, it´s not clustered or load balanced. Each server is dedicated for an media type. I do not know, how this can be established in OM. But maybe this is not that complex. However the traffic needs to be redirected somehow, or for each media type a own port is needed. Probably a clustering is much more complex. NGINX is good for balancing maybe, I think also for udp traffic. But clustering and balancing can only work for one room, so I guess It won´t help if one room has much users. The setup will be very complex, error searching really difficult. Gerald Von: Denis Noctor [mailto:[email protected]] Gesendet: Montag, 23. November 2020 22:44 An: [email protected] Betreff: Re: Users per room client browser scalability Thanks for that Maxim. I came across this... https://docs.bigbluebutton.org/2.2/customize.html (scroll down to "Run three parallel Kurento media servers"). This appears to suggest that 3 KMS (audio, video and screesharing) processes on one server. Maybe this could bring us one step further? All the best. Denis On Mon, Nov 23, 2020 at 9:31 AM Maxim Solodovnik <[email protected]<mailto:[email protected]>> wrote: I just have fixed "refresh" button So audio/video stream can be "refreshed" without page reload both user-list and video-pod buttons are works according to my tests On Mon, 23 Nov 2020 at 17:08, Peter Dähn <[email protected]<mailto:[email protected]>> wrote: Hi there, this might be because BBB uses 3+ KMS servers clustered (I was unable to find direct link regarding it ...) as far as I know, they use freeswitch for audio and kms just for the video streams... BR Peter Am 23.11.20 um 10:53 schrieb Denis Noctor: Thanks a lot Maxim. I am happy to help in anyway for future tests. Sent from my iPhone On Nov 23, 2020, at 3:43 AM, Maxim Solodovnik <[email protected]<mailto:[email protected]>> wrote: Thanks for very detailed report Denis couple of thoughts inline: On Mon, 23 Nov 2020 at 13:59, Denis Noctor <[email protected]<mailto:[email protected]>> wrote: Hi there everyone, My sincerest apologies for only getting back to you now. As I had mentioned in my previous email that I was going through a personal issue and it took longer than I had anticipated to get back in touch. However, as mentioned before, I have been keeping up to date by reading all the emails in the forum over the last few weeks... and some direct emails also. I also apologize for the length of the email... so maybe you should grab a coffee. As mentioned in my previous email, I set up 10 devices to connect to Room 7 on the OM demo server - all of which where running the latest version of Chrome. 3 machines running Windows 10, 1 running Windows 8, 4 running Windows 7, 2 Amazon Fires (set up to run Chrome) with varying degrees of ram (2, 4, 8 and 12 gigs) The results are as follows: After logging in around 8.22pm (Mexico time), 12th November (OpenMeetings - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build date: 2020-11-09T14:57:23Z , I gradually added other devices to the room. I got to 8. There was a little but of a time lapse... in the sense that I would move from one computer to another... and could still see myself in one feed after I had move to another. It is important to note that 2 of the computers (older HP's) have a slight webcam issue... (I think there is a fauly cable... sometimes works sometimes doesn't - but audio/mic was working fine). When I added 2 the last two devices, things started to break down. The audio quality was clearly reduced... there was a lot of crackling sounds... and some of the users video pods disappeared from some of the devices´screens.... or "empty" video pods filled some screens... on some devices.... but were viewable on others. Some users appeared to be disconnected, though they could continue to view the whiteboards... but had their audio and video disconnected (icons in orange)... when they tried to reconnect... they couldn't... they clicked on the audio / vid icons but with no effect... refreshing the screen sometimes seemed to correct this. While 8 users seemed to be able to connect okay... there was a little bit of a time delay. As you can understand, I don't have headphones and microphones for each and every computer... so I spaced them around my house... when I talked... I could hear my voice being repeated... (I am not referring to echo feedback).... there was a slight time delay by a couple of seconds on some of the devices... moving from device to device. However, with just 5 users in a room, this was not really an issue. From time to time users experienced other users being disconnected or whereby they could see the "empty video pod" with the green border flashing on and off as someone spoke.... but again no audio or video being received.... but it was possible to see the same users on other devices. Users would try to "refresh" the page... again only having access to 4 - 5 users on the page.... and not necessarily seeing the moderator. I finished testing around 9.50pm. Some additional observations: Based on some of the emails over the last few weeks. It appears to be that one OM instance can only deal with 3 simultaneous rooms with 5 users approx in each room (using audio and video)... and based on the above maybe a little more, but at a stretch. This appears to boil down to limitations due to number Kurento / WebRTC connections.... some of you have mention somewhere in the range of 200 - 300 connections. As a result I took a look at a few sites regarding BigBlueButton (BBB), as it also uses Kurento and WebRTC to get a general idea as to how many users can be in a room (with camera and audio). However, a lot of digging had to be done as many of the numbers that are used are about how many participants can be in a room (without cam and mic) with a moderator (using cam and mic). Now I apologize for bringing up BBB in conversation, as I am not endorsing the platform....reminding me of Harry Potter, (Voldemort) "He who shall not be named" :) However, it might be worth investigating for ideas on how to increase the number of cams / mics in an OM room. You can view this information here: https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- Similarly, if you look at the following link it suggests that the more number of concurrent users... the number of rooms will be less (but again not taking into consideration cams and mics) : https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support However, regarding the limited number of users (with cam and mic), BBB seems to have got arround this by having a window of 5 cams that can be scrolled left or right. It appears moderators can still view up to 25 cams etc. Take a look at this: https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates this might be because BBB uses 3+ KMS servers clustered (I was unable to find direct link regarding it ...) I guess we might contact BBB devs and ask for the help in KMS configuration/clustering _________________________ OM Demo Next Server Specs: @Maxim, is it possible to share the specs of the demo server, ram, cpu, etc, so that we can get a general idea as to what to benchmark against. I did share the specs before here they are dedicated server with CPU: 8x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz RAM: 24GB It hosting om-demo: 8GB for java om-next: 4GB for java These instances are using - same KMS - same Coturn - Separate databases at same MySql I can increase java memory for demo-next if you are planning to do tests one more time At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). However, I am under the impression that even if I upgrade my AWS server... I am not really going to see any major improvements given the fact that I am still limited to the number of users (with cam and mic) per room. Before this whole covid situation, my school had 10 classrooms, class size average 10. I would like to be able to have a similar virtual set up... but based on all the info above I would need 3 - 4 instances (using clustering, which I have never done before) - but will still have a problem having 11 users (10 students / 1 moderator) in a room. And while I know the following question (as an alternative) might be considered ridiculous - can a multiple number of OM installations (with multiple KMS etc) be conducted in one server - which more ram, cpu power etc? It would be great if anyone out there has a successful clustering model that they could share - even to test across 2 instances. At present, I am using Ubuntu 18.04 on AWS as described above. Either way it seems the main obstacle at the moment seems to be how Kurento and WebRTC can be set up to overcome these limitations Apologies once again for the length of this email and for taking so long to get back in touch. (I've added a few screenshots regarding my test below (one computer's time is 2 hours behind for some strange reason) :)) All the best, Denis. Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik <[email protected]> wrote:<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> On Fri, 13 Nov 2020 at 14:24, Denis Noctor <[email protected]> wrote:<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Hi there Maxim... I did a test with 8 computers and 2 tablets last night (spread across 2 WiFis)... please don’t delete the logs on the OM demo server (next)...<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> I just backup the logs<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> I will come back to you all with some feedback and pics later tomorrow (if that’s okay)... however, for reference... I started the process in the public room #7...start time around 8.22pm (12th Nov) (México... 6 hrs behind) and end time 9.50pm... (if you want to check the logs) .... the short version is that 8 users experienced relatively stable performance. <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Looking forward to hear the full version :))<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Will give you a more detailed feedback once I deal with a personal issue. All the best, Denis.<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Sent from my iPhone<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <[email protected]> wrote:<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Hello All, <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> I'll try to answer in one email :)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> On Wed, 11 Nov 2020 at 20:32, dww <[email protected]> wrote:<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> However, Denis, I think your experiment with multiple devices would be valuable as then there is only one browser tab or window with the OM room open as a guest on each device. Perhaps that will make a difference.<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> yes, this would be better test (even if "fake" camera is used)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Dennis On Wed, 2020-11-11 at 08:24 -0500, dww wrote: > Thanks, Denis, > > Back on Oct. 17 Maxim provided the following Bash script to be run on > the machine with a client side browser for the psuedo guest users. ( > Use another machine to create the room administratively and send > invitations) This is a far simpler way to stress test the client side > browser. > > Dennis > > Hello, > > i just have tried the following script > started as `./run10.sh 5` > > everything seems to work, but my CPU was 800% busy (all cores were > 100% > busy) > > without `--use-fake-device-for-media-stream` parameter I had lots of > permission errors due to camera was "captured" by first browser > other have reported "Camera busy" error > > > _HASH_HERE_ - should be replaced with real hash (I have created > endless > invitation hash to the private conference room) > > the script > =============================================== > #!/bin/bash > > i=$1 > > if [ -z "${i}" ]; then > i=30 > fi > let "i += 0" > > rm -rf /tmp/delme* > > while ((i--)); do > #echo "${i}" > mkdir /tmp/delme${i} > > #local conference > chromium-browser --user-data-dir=/tmp/delme${i} --disable-infobars > --no-default-browser-check --allow-insecure-localhost > --use-fake-device-for-media-stream ' > https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1' > & > done > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: > > Hi there everyone, this seems to be the “elephant in the room” > > discussion, while there has been a HUGE amount of development and > > progress in OM since March (thank you so much @Maxim) ... there is > > the whole issue of, for example, the number of users per room... > > which seems to be about 5-6 (and maybe even to 7) when pushed to > > the > > limit... with both audio and video being broadcasted from all > > users... and, something else.. if there are simultaneous > > classes/sessions being held on the same server... will this > > restrict > > things even further? Is this an overall limitation > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Sebastian did some AWS based testing<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> And, if i'm not mistaken, the server with 4GB RAM was able to handle at least 3 rooms of 5 people<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> (5.1.0-SNAPSHOT should behave better than 5.0.1)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> to increase the number of rooms you can use cluster<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> to using a > > browser > > based approach... or should we be taking > > approach?<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> well,<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> there is "The Limit"<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> KMS can handle only certain amount of multimedia connections<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> additionally there are other limits:<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> - bandwidth<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> - CPU<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> - RAM<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> - open files (network socket is a file)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> "The Limit" is something I'm not sure how to deal with (yet)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > It was my intention to test out the OM “demo servers” over the last > > 2 > > weeks but will take today off and try to test 10 real device > > connections... with a combination of desktops, laptops, android > > tablets and maybe even the odd iPhone or > > two.<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Apple devices has issues with sound (outgoing)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> I'm still investigating this one<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > My million dollar question is... prior to WebRTC and Kurento... was > > it possible to have 5-10 users in a room with audio and video > > working > > seamlessly in previous versions (for example, the old “flash” setup > > (which will be redundant after Christmas... Chrome etc > > notifications) > > and if so, what has > > changed?<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Yes this was possible<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> OM_before_5 was based on Red5 media server<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Unfortunately it's open source version has no WebRTC support<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > If there is anyone out there that has no problem with user numbers > > (using audio and vid)... exceeding a body of 7-10+, please let us > > know. > > > > In the meantime, I’ll give you my feedback on my tests. > > > > I really appreciate everything that has been done to date. > > > > Thanks. > > > > Sent from my iPhone > > > > > On Nov 9, 2020, at 4:50 PM, dww <[email protected]> wrote: > > > > > > Hello Maxim, > > > > > > A couple of weeks ago there was an email thread about the 5 total > > > users > > > for one room, each user with video/microphone under the > > > Subject: "docker container clustering experiments #1". > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> For whatever reason you love to start new mail threads :))))<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> In this > > > case > > > it > > > appears the bottleneck is the CPU usage on the client machine > > > with > > > the > > > browser. > > > > > > In a response to Denis Noctor on a similar thread you mentioned > > > to > > > try > > > the following: > > > > > > "please check allowed amount of opened files for the user who > > > starts > > > OM/KMS/TURN > > > increasing it might help" > > > > > > Might this help with the issue we discussed? Where approximately > > > do > > > I > > > set the allowed amount of opened > > > files?<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> KMS seems to drop connections when there is not enough files<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> (network socket is a file)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> you can check the limit for current user using `ulimit -n` (`ulimit -a` to see all limits)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> to check limit for `nobody` user `su nobody --shell /bin/bash --command "ulimit -n"`<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> to increase the limit i'm changing `/etc/security/limits.conf` file<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> (seems to work at Ubuntu)<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > Also are there any other things that can be tried to improve this > > > scalability? Are there areas in the code that can be examined to > > > investigate how to improve > > > this?<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> KMS cluster would be ultimate solution, I guess<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > > > > > > Thanks, > > > Dennis > > > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> -- <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Best regards, Maxim<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> -- <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Best regards, Maxim<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> -- <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Best regards, Maxim<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> -- <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Best regards, Maxim<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
