https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L74
maybe we need to compare latest default with our version :) On Tue, 2 Feb 2021 at 16:13, [email protected] <[email protected]> wrote: > It says default maxThreads is 200 > https://tomcat.apache.org/tomcat-9.0-doc/config/executor.html > > So I can try with 400 maybe to double that. > > Thanks > Seb > > Sebastian Wagner > Director Arrakeen Solutions, OM-Hosting.com > http://arrakeen-solutions.co.nz/ > https://om-hosting.com - Cloud & Server Hosting for HTML5 > Video-Conferencing OpenMeetings > < > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > > > < > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > > > > > On Tue, 2 Feb 2021 at 22:11, [email protected] <[email protected]> > wrote: > > > I will have a look with 300 and repeat it. > > > > > > BTW are you using dockerized OM? how are you passing `xmx` via > > CATALINA_OPTS > > ? > > => I have a custom Openmeetings docker container and I set those via > > CATALINA_OPS that are passed into the OpenMeetings instance. > > I can see in the cataline.out logs that it reads the values in and uses > it. > > > > Are you setting additional memory for docker? > > => The Docker container itself also has 4GB memory available. > > > > If you compare the graphs from the 2GB and 4GB test you can see that > > memory usage in % has dropped by exactly 50%. So it seems pretty > convincing > > that those settings are all correctly applied. > > > > Thanks > > Seb > > > > Sebastian Wagner > > Director Arrakeen Solutions, OM-Hosting.com > > http://arrakeen-solutions.co.nz/ > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > > Video-Conferencing OpenMeetings > > > > < > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > > > > < > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > > > > > > > > On Tue, 2 Feb 2021 at 22:04, Maxim Solodovnik <[email protected]> > > wrote: > > > >> the default is 150 > >> could you set to 300? > >> we will see is there will be improvement > >> > >> BTW are you using dockerized OM? how are you passing `xmx` via > >> CATALINA_OPTS > >> ? > >> Are you setting additional memory for docker? > >> > >> On Tue, 2 Feb 2021 at 16:00, [email protected] < > [email protected] > >> > > >> wrote: > >> > >> > I can try and re-run, how many would you recommend worth trying for > this > >> > scenario ? > >> > > >> > Thanks > >> > Seb > >> > > >> > Sebastian Wagner > >> > Director Arrakeen Solutions, OM-Hosting.com > >> > http://arrakeen-solutions.co.nz/ > >> > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > Video-Conferencing OpenMeetings > >> > < > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > >> > < > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > >> > > >> > > >> > On Tue, 2 Feb 2021 at 21:56, Maxim Solodovnik <[email protected]> > >> > wrote: > >> > > >> > > Have you tried to increase maxThreads for Tomcat? > >> > > > >> > > On Tue, 2 Feb 2021 at 15:26, [email protected] < > >> > [email protected]> > >> > > wrote: > >> > > > >> > > > I doubled it to 4GB OpenMeetings and 4GB KMS. I updated the docker > >> > > instance > >> > > > to run Openmeetings with xms=2GB and Xmx=4GB. > >> > > > > >> > > > And I did run exactly the same test again: > >> > > > - 50-60 users > >> > > > - staggered to enter in a time period around 5-10min > >> > > > - distributed into 10 conference rooms 4x4 and 2 webinars with 20 > >> > users > >> > > > each > >> > > > - each test runs calls the API to login/createRoomHash and then > >> load > >> > the > >> > > > URL with the room (plus start webcam/audio stream in the > conference > >> > > rooms) > >> > > > > >> > > > The results look almost the same. There is hardly any improvement: > >> > > > > >> > > > - CPU still spikes to almost 100%, memory is not a problem > >> > > > - Empty video pods as well as video pods where webcam stream > >> didn't > >> > > > start > >> > > > > >> > > > There isn't a crash, but that is mostly because I stagger it to > >> enter > >> > the > >> > > > server over a 5-10min period. Which didn't crash the 2GB instance > >> > either. > >> > > > > >> > > > Comparison of the CPU graphs of both hardware configuration and > test > >> > > runs: > >> > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > > >> > > > There is pretty much no improvement. > >> > > > > >> > > > There is some work on the application side needed. This does not > >> look > >> > > like > >> > > > getting better by throwing more hardware at it. > >> > > > > >> > > > It is really quite limiting to have no logs about any sort of > >> > performance > >> > > > indicators like call length to narrow down where the bottleneck > is. > >> > > > You may find some very low hanging fruits in terms of optimisation > >> if > >> > you > >> > > > can simply concentrate on the top ten calls and optimise those. > >> > > > Rather than looking at CPU and memory graphs. > >> > > > > >> > > > Thanks > >> > > > Sebastian > >> > > > > >> > > > Sebastian Wagner > >> > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > http://arrakeen-solutions.co.nz/ > >> > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > Video-Conferencing OpenMeetings > >> > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > >> > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > >> > > > > >> > > > > >> > > > On Tue, 2 Feb 2021 at 17:18, [email protected] < > >> > > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Have we ever looked into which java method would require the > most > >> > > > > resources/time during the process of entering the conference > room > >> ? > >> > > > > > >> > > > > Sebastian Wagner > >> > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > http://arrakeen-solutions.co.nz/ > >> > > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > > Video-Conferencing OpenMeetings > >> > > > > > >> > > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > >> > > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Tue, 2 Feb 2021 at 16:48, Maxim Solodovnik < > >> [email protected]> > >> > > > > wrote: > >> > > > > > >> > > > >> While do load testing I did the following: > >> > > > >> > >> > > > >> create Jmeter test loading "semistatic" stateless error page > with > >> > 300 > >> > > > >> simultaneous threads (I can share this test it is very simple) > >> > > > >> CPU usage of OM process was near to 100% > >> > > > >> the situation is better if Tomcat has more threads (maxThread > >> > > parameter) > >> > > > >> > >> > > > >> I guess we need to check "The Ultimate Tomcat Performace Guide" > >> :))) > >> > > > >> > >> > > > >> On Tue, 2 Feb 2021 at 10:41, [email protected] < > >> > > > [email protected] > >> > > > >> > > >> > > > >> wrote: > >> > > > >> > >> > > > >> > Also the spikes are on the CPU actually more than on the > >> memory: > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > >> > > >> > > > >> > The spike is just 50-60 users. > >> > > > >> > > >> > > > >> > Why would CPU spike to almost 100% just for that amount of > >> users ? > >> > > > >> > > >> > > > >> > I can try with 4GB for Openmeetings and repeat the test. > >> > > > >> > > >> > > > >> > Thanks > >> > > > >> > Seb > >> > > > >> > > >> > > > >> > Sebastian Wagner > >> > > > >> > Director Arrakeen Solutions, OM-Hosting.com > >> > > > >> > http://arrakeen-solutions.co.nz/ > >> > > > >> > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > >> > Video-Conferencing OpenMeetings > >> > > > >> > < > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > >> > > > >> > > > >> > < > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > >> > > > >> > > > >> > > >> > > > >> > > >> > > > >> > On Tue, 2 Feb 2021 at 16:34, Maxim Solodovnik < > >> > [email protected] > >> > > > > >> > > > >> > wrote: > >> > > > >> > > >> > > > >> > > On Tue, 2 Feb 2021 at 10:30, [email protected] < > >> > > > >> > [email protected]> > >> > > > >> > > wrote: > >> > > > >> > > > >> > > > >> > > > I think what you mean is you have OpenMeetings and MySQL > >> and > >> > KMS > >> > > > on > >> > > > >> one > >> > > > >> > > > instance with 4GB. > >> > > > >> > > > > >> > > > >> > > > But its 2GB Just for OpenMeetings. > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > I mean > >> > > > >> > > 4GB just for OM (demo-next) > >> > > > >> > > 8GB just for OM (demo-prod) > >> > > > >> > > and this might need to be increased in case of many users > >> > > > >> > > > >> > > > >> > > Additionally Tomcat's maxThreads might need to be increased > >> > here: > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L74 > >> > > > >> > > > >> > > > >> > > I suspect lot's of simultaneous users need more resources > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > KMS is separated with another 2GB > >> > > > >> > > > MySQL is on another server with another 2GB > >> > > > >> > > > So that would be 6GB in total. But only 2 are allocated > to > >> > > > >> > OpenMeetings. > >> > > > >> > > > > >> > > > >> > > > XmX=2GB for OpenMeetings should be enough and not crash > >> with > >> > > 50-60 > >> > > > >> > users > >> > > > >> > > > entering the room at the same time. > >> > > > >> > > > > >> > > > >> > > > Thanks > >> > > > >> > > > Sebastian > >> > > > >> > > > > >> > > > >> > > > Sebastian Wagner > >> > > > >> > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > >> > > > http://arrakeen-solutions.co.nz/ > >> > > > >> > > > https://om-hosting.com - Cloud & Server Hosting for > HTML5 > >> > > > >> > > > Video-Conferencing OpenMeetings > >> > > > >> > > > < > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > >> > > > > > >> > > > >> > > > < > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > On Tue, 2 Feb 2021 at 16:26, Maxim Solodovnik < > >> > > > [email protected] > >> > > > >> > > >> > > > >> > > > wrote: > >> > > > >> > > > > >> > > > >> > > > > Hello Sebastian, > >> > > > >> > > > > > >> > > > >> > > > > It seems 2GB of RAM is not enough for OM > >> > > > >> > > > > `OutOfMemoryError: Container killed due to memory > >> > usage` > >> > > > >> > > > > I never use less than 4GB (8-16GB in production) > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > On Tue, 2 Feb 2021 at 09:54, Maxim Solodovnik < > >> > > > >> [email protected]> > >> > > > >> > > > > wrote: > >> > > > >> > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > On Tue, 2 Feb 2021 at 07:23, [email protected] < > >> > > > >> > > > > [email protected]> > >> > > > >> > > > > > wrote: > >> > > > >> > > > > > > >> > > > >> > > > > >> Hi, > >> > > > >> > > > > >> > >> > > > >> > > > > >> I have been conducting a few more performance and > load > >> > > tests > >> > > > >> with > >> > > > >> > > the > >> > > > >> > > > > goal > >> > > > >> > > > > >> of increasing participants to 100++. > >> > > > >> > > > > >> > >> > > > >> > > > > >> The challenge is: > >> > > > >> > > > > >> *If more then 50-60 users dynamically create a room > >> Hash > >> > > > (using > >> > > > >> > > > > Soap/Rest > >> > > > >> > > > > >> API) and use that Hash to enter the conference room > >> CPU > >> > and > >> > > > >> memory > >> > > > >> > > > > spikes > >> > > > >> > > > > >> and server crashes* > >> > > > >> > > > > >> > >> > > > >> > > > > > > >> > > > >> > > > > > Can you share API call sequence? > >> > > > >> > > > > > Maybe we can write JMeter scenario for this? > >> > > > >> > > > > > > >> > > > >> > > > > > server crash is something bad > >> > > > >> > > > > > What is happening? Is it a JVM crash? Or is the > system > >> low > >> > > of > >> > > > >> > > resources > >> > > > >> > > > > > and the kernel kills the trouble-maker? > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > >> *Test scenario observations:* > >> > > > >> > > > > >> - It does not matter if those users try to enter > the > >> > same > >> > > > >> room or > >> > > > >> > > > > >> separate > >> > > > >> > > > > >> rooms. In the above test scenario it's a mix of 4x4 > >> > > > conference > >> > > > >> > rooms > >> > > > >> > > > and > >> > > > >> > > > > >> 20x1 webinars > >> > > > >> > > > > >> - This can be reproduced stable and repetitively > >> > > > >> > > > > >> - The issue starts with API calls taking 10sec++ > and > >> > > getting > >> > > > >> more > >> > > > >> > > > > slower. > >> > > > >> > > > > >> Until the OpenMeetings Tomcat instance crashes > >> > > > >> > > > > >> - The issue also manifests that -BEFORE- the server > >> > > crashes > >> > > > >> you > >> > > > >> > can > >> > > > >> > > > see > >> > > > >> > > > > >> video pods not completing the initialisation in the > >> > > > conference > >> > > > >> > room > >> > > > >> > > > > >> itself. > >> > > > >> > > > > >> For example missing video pods or video pods > without a > >> > > webcam > >> > > > >> > > stream. > >> > > > >> > > > > >> Likely to be linked to slow running API or > web-socket > >> > calls > >> > > > >> > > > > >> => I can deliver data samples or screenshots if > >> required > >> > > via > >> > > > >> our > >> > > > >> > > > > >> confluence > >> > > > >> > > > > >> space. > >> > > > >> > > > > >> > >> > > > >> > > > > >> *Hardware and software:* > >> > > > >> > > > > >> - Server and OpenMeetings Instance is isolated on a > >> > > > separated > >> > > > >> > > > hardware > >> > > > >> > > > > >> and > >> > > > >> > > > > >> has 2GB of memory allocated > >> > > > >> > > > > >> - There is no spike on KMS or Database > >> > > hardware/CPU/memory. > >> > > > >> The > >> > > > >> > > spike > >> > > > >> > > > > is > >> > > > >> > > > > >> only in the OpenMeetings Tomcat Server instance > >> > > > >> > > > > >> > >> > > > >> > > > > >> *Possible ways to mitigate without code changes:* > >> > > > >> > > > > >> - You can mitigate part of this issue if you spread > >> the > >> > > > users > >> > > > >> to > >> > > > >> > > > enter > >> > > > >> > > > > >> over a longer time period. However it needs more > than > >> > 10min > >> > > > >> > > separation > >> > > > >> > > > > to > >> > > > >> > > > > >> enter without issues for 50-60 participants > >> > > > >> > > > > >> - You can mitigate part of this issue if you for > >> example > >> > > > >> create > >> > > > >> > the > >> > > > >> > > > > >> room-hash in a different process (like 1h before > >> using) > >> > and > >> > > > >> once > >> > > > >> > all > >> > > > >> > > > > >> hashes > >> > > > >> > > > > >> are created you enter the conference room. It still > >> leads > >> > > to > >> > > > >> > issues, > >> > > > >> > > > but > >> > > > >> > > > > >> you can enter up to 100 users within 5-10min, if you > >> just > >> > > use > >> > > > >> the > >> > > > >> > > > links, > >> > > > >> > > > > >> rather than create the link AND entering with the > >> link at > >> > > the > >> > > > >> same > >> > > > >> > > > > >> time/process > >> > > > >> > > > > >> - Increasing Tomcat to more than 2GB of memory per > >> > Tomcat > >> > > > >> > instance > >> > > > >> > > > may > >> > > > >> > > > > >> help, not sure by how much though > >> > > > >> > > > > >> > >> > > > >> > > > > >> => I think we should spend further time and propose > >> ways > >> > > to > >> > > > >> get > >> > > > >> > rid > >> > > > >> > > > of > >> > > > >> > > > > >> those spikes. The mitigations are not realistic to > >> really > >> > > be > >> > > > >> able > >> > > > >> > to > >> > > > >> > > > use > >> > > > >> > > > > >> in > >> > > > >> > > > > >> practise. > >> > > > >> > > > > >> > >> > > > >> > > > > >> *My proposal is:* > >> > > > >> > > > > >> There is further analysis needed: > >> > > > >> > > > > >> - Capture all OpenMeetings calls that happen during > >> the > >> > > > create > >> > > > >> > room > >> > > > >> > > > > hash > >> > > > >> > > > > >> and conference room-enter > >> > > > >> > > > > >> - Measure call lengths and any calls during the > >> create > >> > > room > >> > > > >> hash > >> > > > >> > > and > >> > > > >> > > > > >> conference room-enter and specific CPU spikes or > >> memory > >> > > usage > >> > > > >> > based > >> > > > >> > > > on a > >> > > > >> > > > > >> per call basis > >> > > > >> > > > > >> - Eventually get a stack trace or have a profile > >> > available > >> > > > >> that > >> > > > >> > > > exports > >> > > > >> > > > > >> the current in memory objects to review where and > what > >> > > create > >> > > > >> > those > >> > > > >> > > > > spikes > >> > > > >> > > > > >> > >> > > > >> > > > > >> Once a per-call analysis is there it should be a lot > >> more > >> > > > easy > >> > > > >> to > >> > > > >> > > > > pinpoint > >> > > > >> > > > > >> specific issues and propose improvements. > >> > > > >> > > > > >> > >> > > > >> > > > > >> As with all performance optimisation this is likely > to > >> > need > >> > > > >> more > >> > > > >> > > > > >> discussion > >> > > > >> > > > > >> once more detailed data is available. > >> > > > >> > > > > >> > >> > > > >> > > > > >> Thanks, > >> > > > >> > > > > >> Sebastian > >> > > > >> > > > > >> > >> > > > >> > > > > >> Sebastian Wagner > >> > > > >> > > > > >> Director Arrakeen Solutions, OM-Hosting.com > >> > > > >> > > > > >> http://arrakeen-solutions.co.nz/ > >> > > > >> > > > > >> https://om-hosting.com - Cloud & Server Hosting for > >> > HTML5 > >> > > > >> > > > > >> Video-Conferencing OpenMeetings > >> > > > >> > > > > >> < > >> > > > >> > > > > >> > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > >> > > > > >> > > >> > > > >> > > > > >> < > >> > > > >> > > > > >> > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > >> > > > > >> > > >> > > > >> > > > > >> > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > -- > >> > > > >> > > > > > Best regards, > >> > > > >> > > > > > Maxim > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > -- > >> > > > >> > > > > Best regards, > >> > > > >> > > > > Maxim > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > -- > >> > > > >> > > Best regards, > >> > > > >> > > Maxim > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > >> > >> > > > >> -- > >> > > > >> Best regards, > >> > > > >> Maxim > >> > > > >> > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best regards, > >> > > Maxim > >> > > > >> > > >> > >> > >> -- > >> Best regards, > >> Maxim > >> > > > -- Best regards, Maxim
