this test was on demo-next ordinary "presentation" room was used users were given only permission to use microphone
later on all microphones were turned ON and everything seems to work I have asked to re-test and try to increase the number of participants (too see how many connections we can handle ...) will provide more details as soon as I get them :) On Mon, 30 Nov 2020 at 15:21, Denis Noctor <[email protected]> wrote: > Thanks Maxim... I would really love to know more about this. Is there > anyone out there that can help with such a set up? > > Sent from my iPhone > > On Nov 30, 2020, at 2:17 AM, Maxim Solodovnik <[email protected]> > wrote: > > AFAIK there was successful test with 11 users in audio-only room > So multiple KMS servers can help ... > > On Mon, 30 Nov 2020 at 15:13, Denis Noctor <[email protected]> wrote: > >> Hi there guys. I was wondering if any further thought or considerations >> have been given to the whole numbers of user per room (vid cam / audio) and >> number of concurrent rooms per instance / server as it is seems to be one >> of the biggest issue / limitations regarding the platform. Any feedback >> would be much appreciated. >> >> All the best, >> Denis. >> >> Sent from my iPhone >> >> On Nov 25, 2020, at 4:10 PM, Denis Noctor <[email protected]> wrote: >> >> Will check. I know Chrome uses a lot of resources and after your message >> and some googling “chrome meeting uses lot of cpu”.... there are a lot of >> links over the last year reporting this... for a number of different >> browser based meeting platforms. >> >> Sent from my iPhone >> >> On Nov 25, 2020, at 12:43 PM, dww <[email protected]> wrote: >> >> Check the CPU and network on at least one client device >> >> On Wed, 2020-11-25 at 12:32 -0600, Denis Noctor wrote: >> >> I didn’t as I was testing on the open meetings demo next website. >> Probably in the logs if they have not been cleared. If it is possible to >> view that info I could schedule and conduct another test. >> >> Sent from my iPhone >> >> On Nov 25, 2020, at 11:10 AM, dww <[email protected]> wrote: >> >> Denis, >> >> Did you check the CPU usage on the server and on at least one of the >> client devices when you got to 8 devices.? >> >> Thanks, >> Dennis >> >> On Mon, 2020-11-23 at 16:56 -0600, Denis Noctor wrote: >> >> Comment below.... >> >> On Mon, Nov 23, 2020 at 4:13 PM dww <[email protected]> wrote: >> >> It seems to me that there are 2 separate stress points. >> >> 1. Maximum number of KMS streams (this can be addressed with clustering). >> 2. Maximum number of users with video and audio that can showing live on >> the client browsers (This may require addressing layout on the browsers). >> >> >> >> Tablets (Android based with Chrome) - all video pods were visible, with >> the exception of the problems when I exceed 8. I did not test iOS devices >> as mine cannot be update to the latest iOS version / Safari. >> >> However, one of my students (Android phone) (who is in a small group (5)) >> has been able to view everyone's cam with no problem... though I would >> really only recommend a smartphone as a last resort. >> >> Picture below: >> <tablet_view (2).JPG> >> >> >> >> >> >> >> Also when you did your tests were smart phones and tablets allowing fewer >> users showing on the browser compared to laptops and desktops? >> >> Dennis >> >> >> On Mon, 2020-11-23 at 15:43 -0600, Denis Noctor wrote: >> >> Thanks for that Maxim. >> >> I came across this... >> >> https://docs.bigbluebutton.org/2.2/customize.html >> >> (scroll down to "Run three parallel Kurento media servers"). >> >> This appears to suggest that 3 KMS (audio, video and screesharing) >> processes on one server. Maybe this could bring us one step further? >> >> All the best. >> >> Denis >> >> >> >> On Mon, Nov 23, 2020 at 9:31 AM Maxim Solodovnik <[email protected]> >> wrote: >> >> I just have fixed "refresh" button >> So audio/video stream can be "refreshed" without page reload >> both user-list and video-pod buttons are works according to my tests >> >> On Mon, 23 Nov 2020 at 17:08, Peter Dähn <[email protected]> wrote: >> >> Hi there, >> >> *this might be because BBB uses 3+ KMS servers clustered* >> *(I was unable to find direct link regarding it ...)* >> >> as far as I know, they use freeswitch for audio and kms just for the >> video streams... >> >> BR Peter >> >> >> Am 23.11.20 um 10:53 schrieb Denis Noctor: >> >> Thanks a lot Maxim. I am happy to help in anyway for future tests. >> >> Sent from my iPhone >> >> On Nov 23, 2020, at 3:43 AM, Maxim Solodovnik <[email protected]> >> wrote: >> >> Thanks for very detailed report Denis >> >> couple of thoughts inline: >> >> On Mon, 23 Nov 2020 at 13:59, Denis Noctor <[email protected]> wrote: >> >> Hi there everyone, >> >> My sincerest apologies for only getting back to you now. As I had >> mentioned in my previous email that I was going through a personal issue >> and it took longer than I had anticipated to get back in touch. >> >> However, as mentioned before, I have been keeping up to date by reading >> all the emails in the forum over the last few weeks... and some direct >> emails also. >> >> I also apologize for the length of the email... so maybe you should grab >> a coffee. >> >> As mentioned in my previous email, I set up 10 devices to connect to Room >> 7 on the OM demo server - all of which where running the latest version of >> Chrome. 3 machines running Windows 10, 1 running Windows 8, 4 running >> Windows 7, 2 Amazon Fires (set up to run Chrome) with varying degrees of >> ram (2, 4, 8 and 12 gigs) >> >> *The results are as follows:* >> After logging in around 8.22pm (Mexico time), 12th November (OpenMeetings >> - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build date: 2020-11-09T14:57:23Z >> , I gradually added other devices to the room. I got to 8. There was a >> little but of a time lapse... in the sense that I would move from one >> computer to another... and could still see myself in one feed after I had >> move to another. It is important to note that 2 of the computers (older >> HP's) have a slight webcam issue... (I think there is a fauly cable... >> sometimes works sometimes doesn't - but audio/mic was working fine). >> >> When I added 2 the last two devices, things started to break down. The >> audio quality was clearly reduced... there was a lot of crackling sounds... >> and some of the users video pods disappeared from some of the >> devices´screens.... or "empty" video pods filled some screens... on some >> devices.... but were viewable on others. Some users appeared to be >> disconnected, though they could continue to view the whiteboards... but had >> their audio and video disconnected (icons in orange)... when they tried to >> reconnect... they couldn't... they clicked on the audio / vid icons but >> with no effect... refreshing the screen sometimes seemed to correct this. >> >> While 8 users seemed to be able to connect okay... there was a little bit >> of a time delay. As you can understand, I don't have headphones and >> microphones for each and every computer... so I spaced them around my >> house... when I talked... I could hear my voice being repeated... (I am not >> referring to echo feedback).... there was a slight time delay by a couple >> of seconds on some of the devices... moving from device to device. However, >> with just 5 users in a room, this was not really an issue. >> >> From time to time users experienced other users being disconnected or >> whereby they could see the "empty video pod" with the green border flashing >> on and off as someone spoke.... but again no audio or video being >> received.... but it was possible to see the same users on other devices. >> >> Users would try to "refresh" the page... again only having access to 4 - >> 5 users on the page.... and not necessarily seeing the moderator. I >> finished testing around 9.50pm. >> >> *Some additional observations:* >> >> Based on some of the emails over the last few weeks. It appears to be >> that one OM instance can only deal with 3 simultaneous rooms with 5 users >> approx in each room (using audio and video)... and based on the above maybe >> a little more, but at a stretch. This appears to boil down to limitations >> due to number Kurento / WebRTC connections.... some of you have mention >> somewhere in the range of 200 - 300 connections. >> >> As a result I took a look at a few sites regarding BigBlueButton (BBB), >> as it also uses Kurento and WebRTC to get a general idea as to how many >> users can be in a room (with camera and audio). However, a lot of digging >> had to be done as many of the numbers that are used are about how many >> participants can be in a room (without cam and mic) with a moderator (using >> cam and mic). Now I apologize for bringing up BBB in conversation, as I am >> not endorsing the platform....reminding me of Harry Potter, (Voldemort) "He >> who shall not be named" :) >> >> However, it might be worth investigating for ideas on how to increase the >> number of cams / mics in an OM room. >> >> You can view this information here: >> >> https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- >> >> Similarly, if you look at the following link it suggests that the more >> number of concurrent users... the number of rooms will be less (but again >> not taking into consideration cams and mics) : >> >> https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support >> >> However, regarding the limited number of users (with cam and mic), BBB >> seems to have got arround this by having a window of 5 cams that can be >> scrolled left or right. It appears moderators can still view up to 25 cams >> etc. >> >> Take a look at this: >> https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates >> >> >> >> this might be because BBB uses 3+ KMS servers clustered >> (I was unable to find direct link regarding it ...) >> >> I guess we might contact BBB devs and ask for the help in KMS >> configuration/clustering >> >> >> >> _________________________ >> >> *OM Demo Next Server Specs:* >> >> @Maxim, is it possible to share the specs of the demo server, ram, cpu, >> etc, so that we can get a general idea as to what to benchmark against. >> >> >> I did share the specs before >> here they are >> >> dedicated server with >> CPU: 8x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz >> RAM: 24GB >> >> It hosting >> om-demo: 8GB for java >> om-next: 4GB for java >> >> These instances are using >> - same KMS >> - same Coturn >> - Separate databases at same MySql >> >> I can increase java memory for demo-next if you are planning to do tests >> one more time >> >> >> >> At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). However, >> I am under the impression that even if I upgrade my AWS server... I am not >> really going to see any major improvements given the fact that I am still >> limited to the number of users (with cam and mic) per room. >> >> Before this whole covid situation, my school had 10 classrooms, class >> size average 10. I would like to be able to have a similar virtual set >> up... but based on all the info above I would need 3 - 4 instances (using >> clustering, which I have never done before) - but will still have a problem >> having 11 users (10 students / 1 moderator) in a room. >> >> And while I know the following question (as an alternative) might be >> considered ridiculous - can a multiple number of OM installations (with >> multiple KMS etc) be conducted in one server - which more ram, cpu power >> etc? >> >> It would be great if anyone out there has a successful clustering model >> that they could share - even to test across 2 instances. At present, I am >> using Ubuntu 18.04 on AWS as described above. >> >> Either way it seems the main obstacle at the moment seems to be how >> Kurento and WebRTC can be set up to overcome these limitations >> >> Apologies once again for the length of this email and for taking so long >> to get back in touch. >> >> (I've added a few screenshots regarding my test below (one computer's >> time is 2 hours behind for some strange reason) :)) >> >> All the best, >> >> Denis. >> >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> Virus-free. >> www.avg.com >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> >> On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik <[email protected]> >> wrote: >> >> >> >> On Fri, 13 Nov 2020 at 14:24, Denis Noctor <[email protected]> wrote: >> >> Hi there Maxim... I did a test with 8 computers and 2 tablets last night >> (spread across 2 WiFis)... please don’t delete the logs on the OM demo >> server (next)... >> >> >> I just backup the logs >> >> >> I will come back to you all with some feedback and pics later tomorrow >> (if that’s okay)... however, for reference... I started the process in the >> public room #7...start time around 8.22pm (12th Nov) (México... 6 hrs >> behind) and end time 9.50pm... (if you want to check the logs) .... the >> short version is that 8 users experienced relatively stable performance. >> >> >> Looking forward to hear the full version :)) >> >> >> Will give you a more detailed feedback once I deal with a personal issue. >> All the best, Denis. >> >> Sent from my iPhone >> >> On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <[email protected]> >> wrote: >> >> Hello All, >> >> I'll try to answer in one email :) >> >> On Wed, 11 Nov 2020 at 20:32, dww <[email protected]> wrote: >> >> However, Denis, I think your experiment with multiple devices would be >> valuable as then there is only one browser tab or window with the OM >> room open as a guest on each device. Perhaps that will make a >> difference. >> >> >> yes, this would be better test (even if "fake" camera is used) >> >> >> >> Dennis >> >> On Wed, 2020-11-11 at 08:24 -0500, dww wrote: >> > Thanks, Denis, >> > >> > Back on Oct. 17 Maxim provided the following Bash script to be run on >> > the machine with a client side browser for the psuedo guest users. ( >> > Use another machine to create the room administratively and send >> > invitations) This is a far simpler way to stress test the client side >> > browser. >> > >> > Dennis >> > >> > Hello, >> > >> > i just have tried the following script >> > started as `./run10.sh 5` >> > >> > everything seems to work, but my CPU was 800% busy (all cores were >> > 100% >> > busy) >> > >> > without `--use-fake-device-for-media-stream` parameter I had lots of >> > permission errors due to camera was "captured" by first browser >> > other have reported "Camera busy" error >> > >> > >> > _HASH_HERE_ - should be replaced with real hash (I have created >> > endless >> > invitation hash to the private conference room) >> > >> > the script >> > =============================================== >> > #!/bin/bash >> > >> > i=$1 >> > >> > if [ -z "${i}" ]; then >> > i=30 >> > fi >> > let "i += 0" >> > >> > rm -rf /tmp/delme* >> > >> > while ((i--)); do >> > #echo "${i}" >> > mkdir /tmp/delme${i} >> > >> > #local conference >> > chromium-browser --user-data-dir=/tmp/delme${i} --disable-infobars >> > --no-default-browser-check --allow-insecure-localhost >> > --use-fake-device-for-media-stream ' >> > >> https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1 >> ' >> > & >> > done >> > >> > >> > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: >> > > Hi there everyone, this seems to be the “elephant in the room” >> > > discussion, while there has been a HUGE amount of development and >> > > progress in OM since March (thank you so much @Maxim) ... there is >> > > the whole issue of, for example, the number of users per room... >> > > which seems to be about 5-6 (and maybe even to 7) when pushed to >> > > the >> > > limit... with both audio and video being broadcasted from all >> > > users... and, something else.. if there are simultaneous >> > > classes/sessions being held on the same server... will this >> > > restrict >> > > things even further? Is this an overall limitation >> >> >> Sebastian did some AWS based testing >> And, if i'm not mistaken, the server with 4GB RAM was able to handle at >> least 3 rooms of 5 people >> (5.1.0-SNAPSHOT should behave better than 5.0.1) >> >> to increase the number of rooms you can use cluster >> >> >> to using a >> > > browser >> > > based approach... or should we be taking approach? >> >> >> well, >> there is "The Limit" >> KMS can handle only certain amount of multimedia connections >> additionally there are other limits: >> - bandwidth >> - CPU >> - RAM >> - open files (network socket is a file) >> >> "The Limit" is something I'm not sure how to deal with (yet) >> >> >> > > >> > > It was my intention to test out the OM “demo servers” over the last >> > > 2 >> > > weeks but will take today off and try to test 10 real device >> > > connections... with a combination of desktops, laptops, android >> > > tablets and maybe even the odd iPhone or two. >> >> >> Apple devices has issues with sound (outgoing) >> I'm still investigating this one >> >> >> > > >> > > My million dollar question is... prior to WebRTC and Kurento... was >> > > it possible to have 5-10 users in a room with audio and video >> > > working >> > > seamlessly in previous versions (for example, the old “flash” setup >> > > (which will be redundant after Christmas... Chrome etc >> > > notifications) >> > > and if so, what has changed? >> >> >> Yes this was possible >> OM_before_5 was based on Red5 media server >> Unfortunately it's open source version has no WebRTC support >> >> >> > > >> > > If there is anyone out there that has no problem with user numbers >> > > (using audio and vid)... exceeding a body of 7-10+, please let us >> > > know. >> > > >> > > In the meantime, I’ll give you my feedback on my tests. >> > > >> > > I really appreciate everything that has been done to date. >> > > >> > > Thanks. >> > > >> > > Sent from my iPhone >> > > >> > > > On Nov 9, 2020, at 4:50 PM, dww <[email protected]> wrote: >> > > > >> > > > Hello Maxim, >> > > > >> > > > A couple of weeks ago there was an email thread about the 5 total >> > > > users >> > > > for one room, each user with video/microphone under the >> > > > Subject: "docker container clustering experiments #1". >> >> >> For whatever reason you love to start new mail threads :)))) >> >> >> In this >> > > > case >> > > > it >> > > > appears the bottleneck is the CPU usage on the client machine >> > > > with >> > > > the >> > > > browser. >> > > > >> > > > In a response to Denis Noctor on a similar thread you mentioned >> > > > to >> > > > try >> > > > the following: >> > > > >> > > > "please check allowed amount of opened files for the user who >> > > > starts >> > > > OM/KMS/TURN >> > > > increasing it might help" >> > > > >> > > > Might this help with the issue we discussed? Where approximately >> > > > do >> > > > I >> > > > set the allowed amount of opened files? >> >> >> KMS seems to drop connections when there is not enough files >> (network socket is a file) >> you can check the limit for current user using `ulimit -n` (`ulimit -a` >> to see all limits) >> >> to check limit for `nobody` user `su nobody --shell /bin/bash --command >> "ulimit -n"` >> >> to increase the limit i'm changing `/etc/security/limits.conf` file >> >> https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35 >> >> (seems to work at Ubuntu) >> >> >> >> > > > >> > > > Also are there any other things that can be tried to improve this >> > > > scalability? Are there areas in the code that can be examined to >> > > > investigate how to improve this? >> >> >> KMS cluster would be ultimate solution, I guess >> >> >> > > > >> > > > Thanks, >> > > > Dennis >> > > > >> > > > >> >> >> >> >> -- >> Best regards, >> Maxim >> >> >> >> >> -- >> Best regards, >> Maxim >> >> >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> Virus-free. >> www.avg.com >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> >> >> >> -- >> Best regards, >> Maxim >> >> >> >> >> > > -- > Best regards, > Maxim > > -- Best regards, Maxim
