I'm not sure. Like I say: I stagger room entry by 5-10min. So it's not really a DDoS surge of users. It's a steady but slow growth.
You might be right, there could be such an issue somewhere. Just quite difficult to find without call statistics rights now. Have we looked into enabling some basic performance logs? If you just have all API/WebSocket invocations logged (configurable) for performance analytics, you can find those kind of issues very easy. Capture them and sort by call length, call numbers, sort by top ten => And you get very quickly to a result. Some of those performance logging frameworks are very easy to enable. You can just annotate methods in Java code. And depending on log settings it will then print those statistics to the log file. Even for example into a format that can be further ingested into Prometheus for performance monitoring and graphing of results. Or for example in case of Prometheus generate a HTTP endpoint that exposes the metrics for generating statistics. See: - https://github.com/prometheus/client_java - https://github.com/prometheus/client_java/blob/master/simpleclient_spring_web/src/main/java/io/prometheus/client/spring/web/PrometheusTimeMethod.java - https://prometheus.github.io/client_java/io/prometheus/client/spring/web/PrometheusTimeMethod.html There might be other alternatives to Prometheus. But it is the current tool most widely supported and it seems with a lot of SDKs, examples and support. If we would have such tools available now I think it would be quite easy to pinpoint the bottlenecks. Doesn't need any JProfiler or Yourkit. Those are useful but the setup is a bit harder and you constantly end up enabling/disabling the profiling. Thanks, Sebastian Sebastian Wagner Director Arrakeen Solutions, OM-Hosting.com http://arrakeen-solutions.co.nz/ https://om-hosting.com - Cloud & Server Hosting for HTML5 Video-Conferencing OpenMeetings <https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url> <https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url> On Tue, 2 Feb 2021 at 22:31, Maxim Solodovnik <[email protected]> wrote: > Previous time I saw such many-users-same-time issues > it was because of too many Ajax requests in room > I have moved lot's of them to WS messages and things get better > > Wicket Ajax requests blocks all pages, maybe further improvements are > required > > > On Tue, 2 Feb 2021 at 16:27, Maxim Solodovnik <[email protected]> > wrote: > > > fair enough :) > > > > On Tue, 2 Feb 2021 at 16:26, [email protected] < > [email protected]> > > wrote: > > > >> I think adding cores at some point will be good. > >> But we need to get to some reasonable user numbers on a single > >> core/reasonable memory. > >> Once those numbers are good => Scale it up. > >> > >> I have a try with the threads and report back. > >> > >> Thanks, > >> Seb > >> > >> Sebastian Wagner > >> Director Arrakeen Solutions, OM-Hosting.com > >> http://arrakeen-solutions.co.nz/ > >> https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> Video-Conferencing OpenMeetings > >> < > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > >> < > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > >> > >> > >> On Tue, 2 Feb 2021 at 22:23, Maxim Solodovnik <[email protected]> > >> wrote: > >> > >> > OK > >> > no cores if it is expensive > >> > > >> > just thought multithreaded application can benefit from multiple cores > >> :) > >> > > >> > On Tue, 2 Feb 2021 at 16:21, [email protected] < > >> [email protected]> > >> > wrote: > >> > > >> > > I don't really want to add more cores. The docker container has > >> exactly 1 > >> > > core just for OpenMeetings. And 4GB memory. > >> > > > >> > > We can try with 2 cores. But the price tags on those improvements > are > >> > > getting into a range of not viable options. Except you improve the > >> > > performance by a factor of 10. > >> > > > >> > > Thanks > >> > > Seb > >> > > > >> > > Sebastian Wagner > >> > > Director Arrakeen Solutions, OM-Hosting.com > >> > > http://arrakeen-solutions.co.nz/ > >> > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > Video-Conferencing OpenMeetings > >> > > < > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > < > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > >> > > > >> > > On Tue, 2 Feb 2021 at 22:13, Maxim Solodovnik <[email protected] > > > >> > > wrote: > >> > > > >> > > > Maybe you can add one more core to OM > >> > > > how many do you have right now? > >> > > > > >> > > > On Tue, 2 Feb 2021 at 16:11, [email protected] < > >> > > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > I will have a look with 300 and repeat it. > >> > > > > > >> > > > > > >> > > > > BTW are you using dockerized OM? how are you passing `xmx` via > >> > > > > CATALINA_OPTS > >> > > > > ? > >> > > > > => I have a custom Openmeetings docker container and I set those > >> via > >> > > > > CATALINA_OPS that are passed into the OpenMeetings instance. > >> > > > > I can see in the cataline.out logs that it reads the values in > and > >> > uses > >> > > > it. > >> > > > > > >> > > > > Are you setting additional memory for docker? > >> > > > > => The Docker container itself also has 4GB memory available. > >> > > > > > >> > > > > If you compare the graphs from the 2GB and 4GB test you can see > >> that > >> > > > memory > >> > > > > usage in % has dropped by exactly 50%. So it seems pretty > >> convincing > >> > > that > >> > > > > those settings are all correctly applied. > >> > > > > > >> > > > > Thanks > >> > > > > Seb > >> > > > > > >> > > > > Sebastian Wagner > >> > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > http://arrakeen-solutions.co.nz/ > >> > > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > > Video-Conferencing OpenMeetings > >> > > > > < > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > >> > > > > < > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > >> > > > > > >> > > > > > >> > > > > On Tue, 2 Feb 2021 at 22:04, Maxim Solodovnik < > >> [email protected]> > >> > > > > wrote: > >> > > > > > >> > > > > > the default is 150 > >> > > > > > could you set to 300? > >> > > > > > we will see is there will be improvement > >> > > > > > > >> > > > > > BTW are you using dockerized OM? how are you passing `xmx` via > >> > > > > > CATALINA_OPTS > >> > > > > > ? > >> > > > > > Are you setting additional memory for docker? > >> > > > > > > >> > > > > > On Tue, 2 Feb 2021 at 16:00, [email protected] < > >> > > > > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > I can try and re-run, how many would you recommend worth > >> trying > >> > for > >> > > > > this > >> > > > > > > scenario ? > >> > > > > > > > >> > > > > > > Thanks > >> > > > > > > Seb > >> > > > > > > > >> > > > > > > Sebastian Wagner > >> > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > > > > Video-Conferencing OpenMeetings > >> > > > > > > < > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > >> > > > > > > < > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tue, 2 Feb 2021 at 21:56, Maxim Solodovnik < > >> > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Have you tried to increase maxThreads for Tomcat? > >> > > > > > > > > >> > > > > > > > On Tue, 2 Feb 2021 at 15:26, [email protected] < > >> > > > > > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > I doubled it to 4GB OpenMeetings and 4GB KMS. I updated > >> the > >> > > > docker > >> > > > > > > > instance > >> > > > > > > > > to run Openmeetings with xms=2GB and Xmx=4GB. > >> > > > > > > > > > >> > > > > > > > > And I did run exactly the same test again: > >> > > > > > > > > - 50-60 users > >> > > > > > > > > - staggered to enter in a time period around 5-10min > >> > > > > > > > > - distributed into 10 conference rooms 4x4 and 2 > webinars > >> > with > >> > > > 20 > >> > > > > > > users > >> > > > > > > > > each > >> > > > > > > > > - each test runs calls the API to login/createRoomHash > >> and > >> > > then > >> > > > > load > >> > > > > > > the > >> > > > > > > > > URL with the room (plus start webcam/audio stream in the > >> > > > conference > >> > > > > > > > rooms) > >> > > > > > > > > > >> > > > > > > > > The results look almost the same. There is hardly any > >> > > > improvement: > >> > > > > > > > > > >> > > > > > > > > - CPU still spikes to almost 100%, memory is not a > >> problem > >> > > > > > > > > - Empty video pods as well as video pods where webcam > >> > stream > >> > > > > > didn't > >> > > > > > > > > start > >> > > > > > > > > > >> > > > > > > > > There isn't a crash, but that is mostly because I > stagger > >> it > >> > to > >> > > > > enter > >> > > > > > > the > >> > > > > > > > > server over a 5-10min period. Which didn't crash the 2GB > >> > > instance > >> > > > > > > either. > >> > > > > > > > > > >> > > > > > > > > Comparison of the CPU graphs of both hardware > >> configuration > >> > and > >> > > > > test > >> > > > > > > > runs: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > > > > > > > >> > > > > > > > > There is pretty much no improvement. > >> > > > > > > > > > >> > > > > > > > > There is some work on the application side needed. This > >> does > >> > > not > >> > > > > look > >> > > > > > > > like > >> > > > > > > > > getting better by throwing more hardware at it. > >> > > > > > > > > > >> > > > > > > > > It is really quite limiting to have no logs about any > >> sort of > >> > > > > > > performance > >> > > > > > > > > indicators like call length to narrow down where the > >> > bottleneck > >> > > > is. > >> > > > > > > > > You may find some very low hanging fruits in terms of > >> > > > optimisation > >> > > > > if > >> > > > > > > you > >> > > > > > > > > can simply concentrate on the top ten calls and optimise > >> > those. > >> > > > > > > > > Rather than looking at CPU and memory graphs. > >> > > > > > > > > > >> > > > > > > > > Thanks > >> > > > > > > > > Sebastian > >> > > > > > > > > > >> > > > > > > > > Sebastian Wagner > >> > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > > > > > https://om-hosting.com - Cloud & Server Hosting for > HTML5 > >> > > > > > > > > Video-Conferencing OpenMeetings > >> > > > > > > > > < > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > > > >> > > > > > > > > < > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > On Tue, 2 Feb 2021 at 17:18, [email protected] < > >> > > > > > > > [email protected]> > >> > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Have we ever looked into which java method would > require > >> > the > >> > > > most > >> > > > > > > > > > resources/time during the process of entering the > >> > conference > >> > > > > room ? > >> > > > > > > > > > > >> > > > > > > > > > Sebastian Wagner > >> > > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > > > > > > https://om-hosting.com - Cloud & Server Hosting for > >> HTML5 > >> > > > > > > > > > Video-Conferencing OpenMeetings > >> > > > > > > > > > > >> > > > > > > > > > < > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > > > >> > > > > > > > > > < > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > On Tue, 2 Feb 2021 at 16:48, Maxim Solodovnik < > >> > > > > > [email protected]> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > >> While do load testing I did the following: > >> > > > > > > > > >> > >> > > > > > > > > >> create Jmeter test loading "semistatic" stateless > error > >> > page > >> > > > > with > >> > > > > > > 300 > >> > > > > > > > > >> simultaneous threads (I can share this test it is > very > >> > > simple) > >> > > > > > > > > >> CPU usage of OM process was near to 100% > >> > > > > > > > > >> the situation is better if Tomcat has more threads > >> > > (maxThread > >> > > > > > > > parameter) > >> > > > > > > > > >> > >> > > > > > > > > >> I guess we need to check "The Ultimate Tomcat > >> Performace > >> > > > Guide" > >> > > > > > :))) > >> > > > > > > > > >> > >> > > > > > > > > >> On Tue, 2 Feb 2021 at 10:41, [email protected] < > >> > > > > > > > > [email protected] > >> > > > > > > > > >> > > >> > > > > > > > > >> wrote: > >> > > > > > > > > >> > >> > > > > > > > > >> > Also the spikes are on the CPU actually more than > on > >> the > >> > > > > memory: > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > > > > > > >> > > >> > > > > > > > > >> > The spike is just 50-60 users. > >> > > > > > > > > >> > > >> > > > > > > > > >> > Why would CPU spike to almost 100% just for that > >> amount > >> > of > >> > > > > > users ? > >> > > > > > > > > >> > > >> > > > > > > > > >> > I can try with 4GB for Openmeetings and repeat the > >> test. > >> > > > > > > > > >> > > >> > > > > > > > > >> > Thanks > >> > > > > > > > > >> > Seb > >> > > > > > > > > >> > > >> > > > > > > > > >> > Sebastian Wagner > >> > > > > > > > > >> > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > > > > > >> > http://arrakeen-solutions.co.nz/ > >> > > > > > > > > >> > https://om-hosting.com - Cloud & Server Hosting > for > >> > HTML5 > >> > > > > > > > > >> > Video-Conferencing OpenMeetings > >> > > > > > > > > >> > < > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > >> > > > >> > > > > > > > > >> > < > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > On Tue, 2 Feb 2021 at 16:34, Maxim Solodovnik < > >> > > > > > > [email protected] > >> > > > > > > > > > >> > > > > > > > > >> > wrote: > >> > > > > > > > > >> > > >> > > > > > > > > >> > > On Tue, 2 Feb 2021 at 10:30, > [email protected] > >> < > >> > > > > > > > > >> > [email protected]> > >> > > > > > > > > >> > > wrote: > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > I think what you mean is you have OpenMeetings > >> and > >> > > MySQL > >> > > > > and > >> > > > > > > KMS > >> > > > > > > > > on > >> > > > > > > > > >> one > >> > > > > > > > > >> > > > instance with 4GB. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > But its 2GB Just for OpenMeetings. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > I mean > >> > > > > > > > > >> > > 4GB just for OM (demo-next) > >> > > > > > > > > >> > > 8GB just for OM (demo-prod) > >> > > > > > > > > >> > > and this might need to be increased in case of > many > >> > > users > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Additionally Tomcat's maxThreads might need to be > >> > > > increased > >> > > > > > > here: > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L74 > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > I suspect lot's of simultaneous users need more > >> > > resources > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > KMS is separated with another 2GB > >> > > > > > > > > >> > > > MySQL is on another server with another 2GB > >> > > > > > > > > >> > > > So that would be 6GB in total. But only 2 are > >> > > allocated > >> > > > to > >> > > > > > > > > >> > OpenMeetings. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > XmX=2GB for OpenMeetings should be enough and > not > >> > > crash > >> > > > > with > >> > > > > > > > 50-60 > >> > > > > > > > > >> > users > >> > > > > > > > > >> > > > entering the room at the same time. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > Thanks > >> > > > > > > > > >> > > > Sebastian > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > Sebastian Wagner > >> > > > > > > > > >> > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > > > > > >> > > > http://arrakeen-solutions.co.nz/ > >> > > > > > > > > >> > > > https://om-hosting.com - Cloud & Server > Hosting > >> for > >> > > > HTML5 > >> > > > > > > > > >> > > > Video-Conferencing OpenMeetings > >> > > > > > > > > >> > > > < > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > < > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > On Tue, 2 Feb 2021 at 16:26, Maxim Solodovnik < > >> > > > > > > > > [email protected] > >> > > > > > > > > >> > > >> > > > > > > > > >> > > > wrote: > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > Hello Sebastian, > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > It seems 2GB of RAM is not enough for OM > >> > > > > > > > > >> > > > > `OutOfMemoryError: Container killed due > >> to > >> > > > memory > >> > > > > > > usage` > >> > > > > > > > > >> > > > > I never use less than 4GB (8-16GB in > >> production) > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > On Tue, 2 Feb 2021 at 09:54, Maxim > Solodovnik < > >> > > > > > > > > >> [email protected]> > >> > > > > > > > > >> > > > > wrote: > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > On Tue, 2 Feb 2021 at 07:23, > >> > > [email protected] > >> > > > < > >> > > > > > > > > >> > > > > [email protected]> > >> > > > > > > > > >> > > > > > wrote: > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > >> Hi, > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> I have been conducting a few more > >> performance > >> > and > >> > > > > load > >> > > > > > > > tests > >> > > > > > > > > >> with > >> > > > > > > > > >> > > the > >> > > > > > > > > >> > > > > goal > >> > > > > > > > > >> > > > > >> of increasing participants to 100++. > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> The challenge is: > >> > > > > > > > > >> > > > > >> *If more then 50-60 users dynamically > >> create a > >> > > room > >> > > > > > Hash > >> > > > > > > > > (using > >> > > > > > > > > >> > > > > Soap/Rest > >> > > > > > > > > >> > > > > >> API) and use that Hash to enter the > >> conference > >> > > room > >> > > > > CPU > >> > > > > > > and > >> > > > > > > > > >> memory > >> > > > > > > > > >> > > > > spikes > >> > > > > > > > > >> > > > > >> and server crashes* > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > Can you share API call sequence? > >> > > > > > > > > >> > > > > > Maybe we can write JMeter scenario for > this? > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > server crash is something bad > >> > > > > > > > > >> > > > > > What is happening? Is it a JVM crash? Or is > >> the > >> > > > system > >> > > > > > low > >> > > > > > > > of > >> > > > > > > > > >> > > resources > >> > > > > > > > > >> > > > > > and the kernel kills the trouble-maker? > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > >> *Test scenario observations:* > >> > > > > > > > > >> > > > > >> - It does not matter if those users try > to > >> > enter > >> > > > the > >> > > > > > > same > >> > > > > > > > > >> room or > >> > > > > > > > > >> > > > > >> separate > >> > > > > > > > > >> > > > > >> rooms. In the above test scenario it's a > >> mix of > >> > > 4x4 > >> > > > > > > > > conference > >> > > > > > > > > >> > rooms > >> > > > > > > > > >> > > > and > >> > > > > > > > > >> > > > > >> 20x1 webinars > >> > > > > > > > > >> > > > > >> - This can be reproduced stable and > >> > repetitively > >> > > > > > > > > >> > > > > >> - The issue starts with API calls taking > >> > 10sec++ > >> > > > and > >> > > > > > > > getting > >> > > > > > > > > >> more > >> > > > > > > > > >> > > > > slower. > >> > > > > > > > > >> > > > > >> Until the OpenMeetings Tomcat instance > >> crashes > >> > > > > > > > > >> > > > > >> - The issue also manifests that -BEFORE- > >> the > >> > > > server > >> > > > > > > > crashes > >> > > > > > > > > >> you > >> > > > > > > > > >> > can > >> > > > > > > > > >> > > > see > >> > > > > > > > > >> > > > > >> video pods not completing the > >> initialisation in > >> > > the > >> > > > > > > > > conference > >> > > > > > > > > >> > room > >> > > > > > > > > >> > > > > >> itself. > >> > > > > > > > > >> > > > > >> For example missing video pods or video > pods > >> > > > without > >> > > > > a > >> > > > > > > > webcam > >> > > > > > > > > >> > > stream. > >> > > > > > > > > >> > > > > >> Likely to be linked to slow running API or > >> > > > web-socket > >> > > > > > > calls > >> > > > > > > > > >> > > > > >> => I can deliver data samples or > >> screenshots if > >> > > > > > required > >> > > > > > > > via > >> > > > > > > > > >> our > >> > > > > > > > > >> > > > > >> confluence > >> > > > > > > > > >> > > > > >> space. > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> *Hardware and software:* > >> > > > > > > > > >> > > > > >> - Server and OpenMeetings Instance is > >> isolated > >> > > on > >> > > > a > >> > > > > > > > > separated > >> > > > > > > > > >> > > > hardware > >> > > > > > > > > >> > > > > >> and > >> > > > > > > > > >> > > > > >> has 2GB of memory allocated > >> > > > > > > > > >> > > > > >> - There is no spike on KMS or Database > >> > > > > > > > hardware/CPU/memory. > >> > > > > > > > > >> The > >> > > > > > > > > >> > > spike > >> > > > > > > > > >> > > > > is > >> > > > > > > > > >> > > > > >> only in the OpenMeetings Tomcat Server > >> instance > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> *Possible ways to mitigate without code > >> > changes:* > >> > > > > > > > > >> > > > > >> - You can mitigate part of this issue if > >> you > >> > > > spread > >> > > > > > the > >> > > > > > > > > users > >> > > > > > > > > >> to > >> > > > > > > > > >> > > > enter > >> > > > > > > > > >> > > > > >> over a longer time period. However it > needs > >> > more > >> > > > than > >> > > > > > > 10min > >> > > > > > > > > >> > > separation > >> > > > > > > > > >> > > > > to > >> > > > > > > > > >> > > > > >> enter without issues for 50-60 > participants > >> > > > > > > > > >> > > > > >> - You can mitigate part of this issue if > >> you > >> > for > >> > > > > > example > >> > > > > > > > > >> create > >> > > > > > > > > >> > the > >> > > > > > > > > >> > > > > >> room-hash in a different process (like 1h > >> > before > >> > > > > using) > >> > > > > > > and > >> > > > > > > > > >> once > >> > > > > > > > > >> > all > >> > > > > > > > > >> > > > > >> hashes > >> > > > > > > > > >> > > > > >> are created you enter the conference room. > >> It > >> > > still > >> > > > > > leads > >> > > > > > > > to > >> > > > > > > > > >> > issues, > >> > > > > > > > > >> > > > but > >> > > > > > > > > >> > > > > >> you can enter up to 100 users within > >> 5-10min, > >> > if > >> > > > you > >> > > > > > just > >> > > > > > > > use > >> > > > > > > > > >> the > >> > > > > > > > > >> > > > links, > >> > > > > > > > > >> > > > > >> rather than create the link AND entering > >> with > >> > the > >> > > > > link > >> > > > > > at > >> > > > > > > > the > >> > > > > > > > > >> same > >> > > > > > > > > >> > > > > >> time/process > >> > > > > > > > > >> > > > > >> - Increasing Tomcat to more than 2GB of > >> memory > >> > > per > >> > > > > > > Tomcat > >> > > > > > > > > >> > instance > >> > > > > > > > > >> > > > may > >> > > > > > > > > >> > > > > >> help, not sure by how much though > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> => I think we should spend further time > and > >> > > > propose > >> > > > > > ways > >> > > > > > > > to > >> > > > > > > > > >> get > >> > > > > > > > > >> > rid > >> > > > > > > > > >> > > > of > >> > > > > > > > > >> > > > > >> those spikes. The mitigations are not > >> realistic > >> > > to > >> > > > > > really > >> > > > > > > > be > >> > > > > > > > > >> able > >> > > > > > > > > >> > to > >> > > > > > > > > >> > > > use > >> > > > > > > > > >> > > > > >> in > >> > > > > > > > > >> > > > > >> practise. > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> *My proposal is:* > >> > > > > > > > > >> > > > > >> There is further analysis needed: > >> > > > > > > > > >> > > > > >> - Capture all OpenMeetings calls that > >> happen > >> > > > during > >> > > > > > the > >> > > > > > > > > create > >> > > > > > > > > >> > room > >> > > > > > > > > >> > > > > hash > >> > > > > > > > > >> > > > > >> and conference room-enter > >> > > > > > > > > >> > > > > >> - Measure call lengths and any calls > during > >> > the > >> > > > > create > >> > > > > > > > room > >> > > > > > > > > >> hash > >> > > > > > > > > >> > > and > >> > > > > > > > > >> > > > > >> conference room-enter and specific CPU > >> spikes > >> > or > >> > > > > memory > >> > > > > > > > usage > >> > > > > > > > > >> > based > >> > > > > > > > > >> > > > on a > >> > > > > > > > > >> > > > > >> per call basis > >> > > > > > > > > >> > > > > >> - Eventually get a stack trace or have a > >> > profile > >> > > > > > > available > >> > > > > > > > > >> that > >> > > > > > > > > >> > > > exports > >> > > > > > > > > >> > > > > >> the current in memory objects to review > >> where > >> > and > >> > > > > what > >> > > > > > > > create > >> > > > > > > > > >> > those > >> > > > > > > > > >> > > > > spikes > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> Once a per-call analysis is there it > should > >> be > >> > a > >> > > > lot > >> > > > > > more > >> > > > > > > > > easy > >> > > > > > > > > >> to > >> > > > > > > > > >> > > > > pinpoint > >> > > > > > > > > >> > > > > >> specific issues and propose improvements. > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> As with all performance optimisation this > is > >> > > likely > >> > > > > to > >> > > > > > > need > >> > > > > > > > > >> more > >> > > > > > > > > >> > > > > >> discussion > >> > > > > > > > > >> > > > > >> once more detailed data is available. > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> Thanks, > >> > > > > > > > > >> > > > > >> Sebastian > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > >> Sebastian Wagner > >> > > > > > > > > >> > > > > >> Director Arrakeen Solutions, > OM-Hosting.com > >> > > > > > > > > >> > > > > >> http://arrakeen-solutions.co.nz/ > >> > > > > > > > > >> > > > > >> https://om-hosting.com - Cloud & Server > >> > Hosting > >> > > > for > >> > > > > > > HTML5 > >> > > > > > > > > >> > > > > >> Video-Conferencing OpenMeetings > >> > > > > > > > > >> > > > > >> < > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> < > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > -- > >> > > > > > > > > >> > > > > > Best regards, > >> > > > > > > > > >> > > > > > Maxim > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > -- > >> > > > > > > > > >> > > > > Best regards, > >> > > > > > > > > >> > > > > Maxim > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > -- > >> > > > > > > > > >> > > Best regards, > >> > > > > > > > > >> > > Maxim > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> -- > >> > > > > > > > > >> Best regards, > >> > > > > > > > > >> Maxim > >> > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > -- > >> > > > > > > > Best regards, > >> > > > > > > > Maxim > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Best regards, > >> > > > > > Maxim > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Best regards, > >> > > > Maxim > >> > > > > >> > > > >> > > >> > > >> > -- > >> > Best regards, > >> > Maxim > >> > > >> > > > > > > -- > > Best regards, > > Maxim > > > > > -- > Best regards, > Maxim >
