80 - is too "beautiful" looking forward to see your further results :)
On Wed, 3 Feb 2021 at 02:21, [email protected] <[email protected]> wrote: > Sry my test numbers are slightly off. > > It’s not 50-60 users, but 80. Exactly 80: > 10 conference rooms 4x4 + 2 webinar rooms with 20 users each = 80 users > > I update the cwiki page. > > I will create a branch to try out adding some performance metrics. To see > how hard but also how useful they will be. I make a Jira for the > investigation. > > Thanks > Seb > > On Tue, 2 Feb 2021 at 11:38 PM, [email protected] < > [email protected]> > wrote: > > > well we need something else (not sure yet how useful it will be) > > => It means you spend less time on running tests for days & weeks and > > re-running them in 100 different configurations in a try-and-error > approach > > without knowing what to look at. > > > > I can point to the demo server, yeah. Not sure if that will do much. > > > > Thanks > > Seb > > > > Sebastian Wagner > > Director Arrakeen Solutions, OM-Hosting.com > > http://arrakeen-solutions.co.nz/ > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > > Video-Conferencing OpenMeetings > > > > < > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > > > > < > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > > > > > > > > On Tue, 2 Feb 2021 at 23:34, Maxim Solodovnik <[email protected]> > > wrote: > > > >> well we need something else (not sure yet how useful it will be) > >> > >> BTW maybe we can use demo-next as server > >> just to compare with your docker? > >> > >> On Tue, 2 Feb 2021 at 17:28, [email protected] < > [email protected] > >> > > >> wrote: > >> > >> > Sorry but access log just counts calls. It doesn't contain call > length. > >> > Which is the thing interesting. > >> > Also as you mentioned: We have a mix of HTTP and websocket calls. > >> > Web-Socket doesn't show up in access logs. > >> > > >> > It's just not useful. You can find out some very basic metrics. But > not > >> > good enough for diagnosing. > >> > > >> > Thanks > >> > Sebastian > >> > > >> > Sebastian Wagner > >> > Director Arrakeen Solutions, OM-Hosting.com > >> > http://arrakeen-solutions.co.nz/ > >> > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > Video-Conferencing OpenMeetings > >> > < > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > >> > < > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > >> > > >> > > >> > On Tue, 2 Feb 2021 at 23:16, Maxim Solodovnik <[email protected]> > >> > wrote: > >> > > >> > > tomcat has accesslog valve > >> > > it should be enabled by default > >> > > > >> > > On Tue, 2 Feb 2021 at 16:49, [email protected] < > >> > [email protected]> > >> > > wrote: > >> > > > >> > > > I'm not sure. Like I say: I stagger room entry by 5-10min. So it's > >> not > >> > > > really a DDoS surge of users. It's a steady but slow growth. > >> > > > > >> > > > You might be right, there could be such an issue somewhere. Just > >> quite > >> > > > difficult to find without call statistics rights now. > >> > > > > >> > > > Have we looked into enabling some basic performance logs? > >> > > > If you just have all API/WebSocket invocations logged > (configurable) > >> > for > >> > > > performance analytics, you can find those kind of issues very > easy. > >> > > > Capture them and sort by call length, call numbers, sort by top > ten > >> => > >> > > And > >> > > > you get very quickly to a result. > >> > > > > >> > > > Some of those performance logging frameworks are very easy to > >> enable. > >> > You > >> > > > can just annotate methods in Java code. And depending on log > >> settings > >> > it > >> > > > will then print those statistics to the log file. > >> > > > Even for example into a format that can be further ingested into > >> > > Prometheus > >> > > > for performance monitoring and graphing of results. Or for example > >> in > >> > > case > >> > > > of Prometheus generate a HTTP endpoint that exposes the metrics > for > >> > > > generating statistics. > >> > > > > >> > > > See: > >> > > > > >> > > > - https://github.com/prometheus/client_java > >> > > > - > >> > > > > >> > > > > >> > > > >> > > >> > https://github.com/prometheus/client_java/blob/master/simpleclient_spring_web/src/main/java/io/prometheus/client/spring/web/PrometheusTimeMethod.java > >> > > > - > >> > > > > >> > > > > >> > > > >> > > >> > https://prometheus.github.io/client_java/io/prometheus/client/spring/web/PrometheusTimeMethod.html > >> > > > > >> > > > There might be other alternatives to Prometheus. But it is the > >> current > >> > > tool > >> > > > most widely supported and it seems with a lot of SDKs, examples > and > >> > > > support. If we would have such tools available now I think it > would > >> be > >> > > > quite easy to pinpoint the bottlenecks. Doesn't need any JProfiler > >> or > >> > > > Yourkit. Those are useful but the setup is a bit harder and you > >> > > constantly > >> > > > end up enabling/disabling the profiling. > >> > > > > >> > > > Thanks, > >> > > > Sebastian > >> > > > > >> > > > Sebastian Wagner > >> > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > http://arrakeen-solutions.co.nz/ > >> > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > Video-Conferencing OpenMeetings > >> > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > > >> > > > < > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > > >> > > > > >> > > > > >> > > > On Tue, 2 Feb 2021 at 22:31, Maxim Solodovnik < > [email protected] > >> > > >> > > > wrote: > >> > > > > >> > > > > Previous time I saw such many-users-same-time issues > >> > > > > it was because of too many Ajax requests in room > >> > > > > I have moved lot's of them to WS messages and things get better > >> > > > > > >> > > > > Wicket Ajax requests blocks all pages, maybe further > improvements > >> are > >> > > > > required > >> > > > > > >> > > > > > >> > > > > On Tue, 2 Feb 2021 at 16:27, Maxim Solodovnik < > >> [email protected]> > >> > > > > wrote: > >> > > > > > >> > > > > > fair enough :) > >> > > > > > > >> > > > > > On Tue, 2 Feb 2021 at 16:26, [email protected] < > >> > > > > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > >> I think adding cores at some point will be good. > >> > > > > >> But we need to get to some reasonable user numbers on a > single > >> > > > > >> core/reasonable memory. > >> > > > > >> Once those numbers are good => Scale it up. > >> > > > > >> > >> > > > > >> I have a try with the threads and report back. > >> > > > > >> > >> > > > > >> Thanks, > >> > > > > >> Seb > >> > > > > >> > >> > > > > >> Sebastian Wagner > >> > > > > >> Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> http://arrakeen-solutions.co.nz/ > >> > > > > >> https://om-hosting.com - Cloud & Server Hosting for HTML5 > >> > > > > >> Video-Conferencing OpenMeetings > >> > > > > >> < > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > >> > > > > >> < > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > >> > > > > >> > >> > > > > >> > >> > > > > >> On Tue, 2 Feb 2021 at 22:23, Maxim Solodovnik < > >> > [email protected] > >> > > > > >> > > > > >> wrote: > >> > > > > >> > >> > > > > >> > OK > >> > > > > >> > no cores if it is expensive > >> > > > > >> > > >> > > > > >> > just thought multithreaded application can benefit from > >> multiple > >> > > > cores > >> > > > > >> :) > >> > > > > >> > > >> > > > > >> > On Tue, 2 Feb 2021 at 16:21, [email protected] < > >> > > > > >> [email protected]> > >> > > > > >> > wrote: > >> > > > > >> > > >> > > > > >> > > I don't really want to add more cores. The docker > container > >> > has > >> > > > > >> exactly 1 > >> > > > > >> > > core just for OpenMeetings. And 4GB memory. > >> > > > > >> > > > >> > > > > >> > > We can try with 2 cores. But the price tags on those > >> > > improvements > >> > > > > are > >> > > > > >> > > getting into a range of not viable options. Except you > >> improve > >> > > the > >> > > > > >> > > performance by a factor of 10. > >> > > > > >> > > > >> > > > > >> > > Thanks > >> > > > > >> > > Seb > >> > > > > >> > > > >> > > > > >> > > Sebastian Wagner > >> > > > > >> > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > https://om-hosting.com - Cloud & Server Hosting for > HTML5 > >> > > > > >> > > Video-Conferencing OpenMeetings > >> > > > > >> > > < > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > >> > > > > >> > > < > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > On Tue, 2 Feb 2021 at 22:13, Maxim Solodovnik < > >> > > > [email protected] > >> > > > > > > >> > > > > >> > > wrote: > >> > > > > >> > > > >> > > > > >> > > > Maybe you can add one more core to OM > >> > > > > >> > > > how many do you have right now? > >> > > > > >> > > > > >> > > > > >> > > > On Tue, 2 Feb 2021 at 16:11, [email protected] < > >> > > > > >> > > [email protected]> > >> > > > > >> > > > wrote: > >> > > > > >> > > > > >> > > > > >> > > > > I will have a look with 300 and repeat it. > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > BTW are you using dockerized OM? how are you passing > >> `xmx` > >> > > via > >> > > > > >> > > > > CATALINA_OPTS > >> > > > > >> > > > > ? > >> > > > > >> > > > > => I have a custom Openmeetings docker container and > I > >> set > >> > > > those > >> > > > > >> via > >> > > > > >> > > > > CATALINA_OPS that are passed into the OpenMeetings > >> > instance. > >> > > > > >> > > > > I can see in the cataline.out logs that it reads the > >> > values > >> > > in > >> > > > > and > >> > > > > >> > uses > >> > > > > >> > > > it. > >> > > > > >> > > > > > >> > > > > >> > > > > Are you setting additional memory for docker? > >> > > > > >> > > > > => The Docker container itself also has 4GB memory > >> > > available. > >> > > > > >> > > > > > >> > > > > >> > > > > If you compare the graphs from the 2GB and 4GB test > you > >> > can > >> > > > see > >> > > > > >> that > >> > > > > >> > > > memory > >> > > > > >> > > > > usage in % has dropped by exactly 50%. So it seems > >> pretty > >> > > > > >> convincing > >> > > > > >> > > that > >> > > > > >> > > > > those settings are all correctly applied. > >> > > > > >> > > > > > >> > > > > >> > > > > Thanks > >> > > > > >> > > > > Seb > >> > > > > >> > > > > > >> > > > > >> > > > > Sebastian Wagner > >> > > > > >> > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> > > > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > https://om-hosting.com - Cloud & Server Hosting for > >> HTML5 > >> > > > > >> > > > > Video-Conferencing OpenMeetings > >> > > > > >> > > > > < > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > >> > > > > >> > > > > < > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > On Tue, 2 Feb 2021 at 22:04, Maxim Solodovnik < > >> > > > > >> [email protected]> > >> > > > > >> > > > > wrote: > >> > > > > >> > > > > > >> > > > > >> > > > > > the default is 150 > >> > > > > >> > > > > > could you set to 300? > >> > > > > >> > > > > > we will see is there will be improvement > >> > > > > >> > > > > > > >> > > > > >> > > > > > BTW are you using dockerized OM? how are you > passing > >> > `xmx` > >> > > > via > >> > > > > >> > > > > > CATALINA_OPTS > >> > > > > >> > > > > > ? > >> > > > > >> > > > > > Are you setting additional memory for docker? > >> > > > > >> > > > > > > >> > > > > >> > > > > > On Tue, 2 Feb 2021 at 16:00, [email protected] > < > >> > > > > >> > > > > [email protected]> > >> > > > > >> > > > > > wrote: > >> > > > > >> > > > > > > >> > > > > >> > > > > > > I can try and re-run, how many would you > recommend > >> > worth > >> > > > > >> trying > >> > > > > >> > for > >> > > > > >> > > > > this > >> > > > > >> > > > > > > scenario ? > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > Thanks > >> > > > > >> > > > > > > Seb > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > Sebastian Wagner > >> > > > > >> > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > https://om-hosting.com - Cloud & Server Hosting > >> for > >> > > HTML5 > >> > > > > >> > > > > > > Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > < > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > < > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > On Tue, 2 Feb 2021 at 21:56, Maxim Solodovnik < > >> > > > > >> > > [email protected]> > >> > > > > >> > > > > > > wrote: > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > Have you tried to increase maxThreads for > Tomcat? > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > On Tue, 2 Feb 2021 at 15:26, > >> [email protected] > >> > < > >> > > > > >> > > > > > > [email protected]> > >> > > > > >> > > > > > > > wrote: > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > I doubled it to 4GB OpenMeetings and 4GB > KMS. I > >> > > > updated > >> > > > > >> the > >> > > > > >> > > > docker > >> > > > > >> > > > > > > > instance > >> > > > > >> > > > > > > > > to run Openmeetings with xms=2GB and Xmx=4GB. > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > And I did run exactly the same test again: > >> > > > > >> > > > > > > > > - 50-60 users > >> > > > > >> > > > > > > > > - staggered to enter in a time period around > >> > > 5-10min > >> > > > > >> > > > > > > > > - distributed into 10 conference rooms 4x4 > >> and 2 > >> > > > > webinars > >> > > > > >> > with > >> > > > > >> > > > 20 > >> > > > > >> > > > > > > users > >> > > > > >> > > > > > > > > each > >> > > > > >> > > > > > > > > - each test runs calls the API to > >> > > > login/createRoomHash > >> > > > > >> and > >> > > > > >> > > then > >> > > > > >> > > > > load > >> > > > > >> > > > > > > the > >> > > > > >> > > > > > > > > URL with the room (plus start webcam/audio > >> stream > >> > in > >> > > > the > >> > > > > >> > > > conference > >> > > > > >> > > > > > > > rooms) > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > The results look almost the same. There is > >> hardly > >> > > any > >> > > > > >> > > > improvement: > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > - CPU still spikes to almost 100%, memory > is > >> > not > >> > > a > >> > > > > >> problem > >> > > > > >> > > > > > > > > - Empty video pods as well as video pods > >> where > >> > > > webcam > >> > > > > >> > stream > >> > > > > >> > > > > > didn't > >> > > > > >> > > > > > > > > start > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > There isn't a crash, but that is mostly > >> because I > >> > > > > stagger > >> > > > > >> it > >> > > > > >> > to > >> > > > > >> > > > > enter > >> > > > > >> > > > > > > the > >> > > > > >> > > > > > > > > server over a 5-10min period. Which didn't > >> crash > >> > the > >> > > > 2GB > >> > > > > >> > > instance > >> > > > > >> > > > > > > either. > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > Comparison of the CPU graphs of both hardware > >> > > > > >> configuration > >> > > > > >> > and > >> > > > > >> > > > > test > >> > > > > >> > > > > > > > runs: > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > There is pretty much no improvement. > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > There is some work on the application side > >> needed. > >> > > > This > >> > > > > >> does > >> > > > > >> > > not > >> > > > > >> > > > > look > >> > > > > >> > > > > > > > like > >> > > > > >> > > > > > > > > getting better by throwing more hardware at > it. > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > It is really quite limiting to have no logs > >> about > >> > > any > >> > > > > >> sort of > >> > > > > >> > > > > > > performance > >> > > > > >> > > > > > > > > indicators like call length to narrow down > >> where > >> > the > >> > > > > >> > bottleneck > >> > > > > >> > > > is. > >> > > > > >> > > > > > > > > You may find some very low hanging fruits in > >> terms > >> > > of > >> > > > > >> > > > optimisation > >> > > > > >> > > > > if > >> > > > > >> > > > > > > you > >> > > > > >> > > > > > > > > can simply concentrate on the top ten calls > and > >> > > > optimise > >> > > > > >> > those. > >> > > > > >> > > > > > > > > Rather than looking at CPU and memory graphs. > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > Thanks > >> > > > > >> > > > > > > > > Sebastian > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > Sebastian Wagner > >> > > > > >> > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> > > > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > > > https://om-hosting.com - Cloud & Server > >> Hosting > >> > for > >> > > > > HTML5 > >> > > > > >> > > > > > > > > Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > > > < > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > < > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > On Tue, 2 Feb 2021 at 17:18, > >> > [email protected] > >> > > < > >> > > > > >> > > > > > > > [email protected]> > >> > > > > >> > > > > > > > > wrote: > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Have we ever looked into which java method > >> would > >> > > > > require > >> > > > > >> > the > >> > > > > >> > > > most > >> > > > > >> > > > > > > > > > resources/time during the process of > entering > >> > the > >> > > > > >> > conference > >> > > > > >> > > > > room ? > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > Sebastian Wagner > >> > > > > >> > > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com > >> > > > > >> > > > > > > > > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > > > > https://om-hosting.com - Cloud & Server > >> Hosting > >> > > for > >> > > > > >> HTML5 > >> > > > > >> > > > > > > > > > Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > < > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > < > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > On Tue, 2 Feb 2021 at 16:48, Maxim > >> Solodovnik < > >> > > > > >> > > > > > [email protected]> > >> > > > > >> > > > > > > > > > wrote: > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > >> While do load testing I did the following: > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> create Jmeter test loading "semistatic" > >> > stateless > >> > > > > error > >> > > > > >> > page > >> > > > > >> > > > > with > >> > > > > >> > > > > > > 300 > >> > > > > >> > > > > > > > > >> simultaneous threads (I can share this > test > >> it > >> > is > >> > > > > very > >> > > > > >> > > simple) > >> > > > > >> > > > > > > > > >> CPU usage of OM process was near to 100% > >> > > > > >> > > > > > > > > >> the situation is better if Tomcat has more > >> > > threads > >> > > > > >> > > (maxThread > >> > > > > >> > > > > > > > parameter) > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> I guess we need to check "The Ultimate > >> Tomcat > >> > > > > >> Performace > >> > > > > >> > > > Guide" > >> > > > > >> > > > > > :))) > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> On Tue, 2 Feb 2021 at 10:41, > >> > > [email protected] > >> > > > < > >> > > > > >> > > > > > > > > [email protected] > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> wrote: > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > Also the spikes are on the CPU actually > >> more > >> > > than > >> > > > > on > >> > > > > >> the > >> > > > > >> > > > > memory: > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > The spike is just 50-60 users. > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > Why would CPU spike to almost 100% just > >> for > >> > > that > >> > > > > >> amount > >> > > > > >> > of > >> > > > > >> > > > > > users ? > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > I can try with 4GB for Openmeetings and > >> > repeat > >> > > > the > >> > > > > >> test. > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > Thanks > >> > > > > >> > > > > > > > > >> > Seb > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > Sebastian Wagner > >> > > > > >> > > > > > > > > >> > Director Arrakeen Solutions, > >> OM-Hosting.com > >> > > > > >> > > > > > > > > >> > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > > > >> > https://om-hosting.com - Cloud & Server > >> > > Hosting > >> > > > > for > >> > > > > >> > HTML5 > >> > > > > >> > > > > > > > > >> > Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > > > >> > < > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > < > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > On Tue, 2 Feb 2021 at 16:34, Maxim > >> > Solodovnik < > >> > > > > >> > > > > > > [email protected] > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > wrote: > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > On Tue, 2 Feb 2021 at 10:30, > >> > > > > [email protected] > >> > > > > >> < > >> > > > > >> > > > > > > > > >> > [email protected]> > >> > > > > >> > > > > > > > > >> > > wrote: > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > I think what you mean is you have > >> > > > OpenMeetings > >> > > > > >> and > >> > > > > >> > > MySQL > >> > > > > >> > > > > and > >> > > > > >> > > > > > > KMS > >> > > > > >> > > > > > > > > on > >> > > > > >> > > > > > > > > >> one > >> > > > > >> > > > > > > > > >> > > > instance with 4GB. > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > But its 2GB Just for OpenMeetings. > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > I mean > >> > > > > >> > > > > > > > > >> > > 4GB just for OM (demo-next) > >> > > > > >> > > > > > > > > >> > > 8GB just for OM (demo-prod) > >> > > > > >> > > > > > > > > >> > > and this might need to be increased in > >> case > >> > > of > >> > > > > many > >> > > > > >> > > users > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > Additionally Tomcat's maxThreads might > >> need > >> > > to > >> > > > be > >> > > > > >> > > > increased > >> > > > > >> > > > > > > here: > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L74 > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > I suspect lot's of simultaneous users > >> need > >> > > more > >> > > > > >> > > resources > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > KMS is separated with another 2GB > >> > > > > >> > > > > > > > > >> > > > MySQL is on another server with > >> another > >> > 2GB > >> > > > > >> > > > > > > > > >> > > > So that would be 6GB in total. But > >> only 2 > >> > > are > >> > > > > >> > > allocated > >> > > > > >> > > > to > >> > > > > >> > > > > > > > > >> > OpenMeetings. > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > XmX=2GB for OpenMeetings should be > >> enough > >> > > and > >> > > > > not > >> > > > > >> > > crash > >> > > > > >> > > > > with > >> > > > > >> > > > > > > > 50-60 > >> > > > > >> > > > > > > > > >> > users > >> > > > > >> > > > > > > > > >> > > > entering the room at the same time. > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > Thanks > >> > > > > >> > > > > > > > > >> > > > Sebastian > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > Sebastian Wagner > >> > > > > >> > > > > > > > > >> > > > Director Arrakeen Solutions, > >> > OM-Hosting.com > >> > > > > >> > > > > > > > > >> > > > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > > > >> > > > https://om-hosting.com - Cloud & > >> Server > >> > > > > Hosting > >> > > > > >> for > >> > > > > >> > > > HTML5 > >> > > > > >> > > > > > > > > >> > > > Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > > > >> > > > < > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > < > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > On Tue, 2 Feb 2021 at 16:26, Maxim > >> > > > Solodovnik < > >> > > > > >> > > > > > > > > [email protected] > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > > wrote: > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > Hello Sebastian, > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > It seems 2GB of RAM is not enough > >> for > >> > OM > >> > > > > >> > > > > > > > > >> > > > > `OutOfMemoryError: Container > >> > killed > >> > > > due > >> > > > > >> to > >> > > > > >> > > > memory > >> > > > > >> > > > > > > usage` > >> > > > > >> > > > > > > > > >> > > > > I never use less than 4GB (8-16GB > in > >> > > > > >> production) > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > On Tue, 2 Feb 2021 at 09:54, Maxim > >> > > > > Solodovnik < > >> > > > > >> > > > > > > > > >> [email protected]> > >> > > > > >> > > > > > > > > >> > > > > wrote: > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > On Tue, 2 Feb 2021 at 07:23, > >> > > > > >> > > [email protected] > >> > > > > >> > > > < > >> > > > > >> > > > > > > > > >> > > > > [email protected]> > >> > > > > >> > > > > > > > > >> > > > > > wrote: > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Hi, > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> I have been conducting a few > more > >> > > > > >> performance > >> > > > > >> > and > >> > > > > >> > > > > load > >> > > > > >> > > > > > > > tests > >> > > > > >> > > > > > > > > >> with > >> > > > > >> > > > > > > > > >> > > the > >> > > > > >> > > > > > > > > >> > > > > goal > >> > > > > >> > > > > > > > > >> > > > > >> of increasing participants to > >> 100++. > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> The challenge is: > >> > > > > >> > > > > > > > > >> > > > > >> *If more then 50-60 users > >> > dynamically > >> > > > > >> create a > >> > > > > >> > > room > >> > > > > >> > > > > > Hash > >> > > > > >> > > > > > > > > (using > >> > > > > >> > > > > > > > > >> > > > > Soap/Rest > >> > > > > >> > > > > > > > > >> > > > > >> API) and use that Hash to enter > >> the > >> > > > > >> conference > >> > > > > >> > > room > >> > > > > >> > > > > CPU > >> > > > > >> > > > > > > and > >> > > > > >> > > > > > > > > >> memory > >> > > > > >> > > > > > > > > >> > > > > spikes > >> > > > > >> > > > > > > > > >> > > > > >> and server crashes* > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > Can you share API call sequence? > >> > > > > >> > > > > > > > > >> > > > > > Maybe we can write JMeter > scenario > >> > for > >> > > > > this? > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > server crash is something bad > >> > > > > >> > > > > > > > > >> > > > > > What is happening? Is it a JVM > >> crash? > >> > > Or > >> > > > is > >> > > > > >> the > >> > > > > >> > > > system > >> > > > > >> > > > > > low > >> > > > > >> > > > > > > > of > >> > > > > >> > > > > > > > > >> > > resources > >> > > > > >> > > > > > > > > >> > > > > > and the kernel kills the > >> > trouble-maker? > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> *Test scenario observations:* > >> > > > > >> > > > > > > > > >> > > > > >> - It does not matter if those > >> users > >> > > try > >> > > > > to > >> > > > > >> > enter > >> > > > > >> > > > the > >> > > > > >> > > > > > > same > >> > > > > >> > > > > > > > > >> room or > >> > > > > >> > > > > > > > > >> > > > > >> separate > >> > > > > >> > > > > > > > > >> > > > > >> rooms. In the above test > scenario > >> > > it's a > >> > > > > >> mix of > >> > > > > >> > > 4x4 > >> > > > > >> > > > > > > > > conference > >> > > > > >> > > > > > > > > >> > rooms > >> > > > > >> > > > > > > > > >> > > > and > >> > > > > >> > > > > > > > > >> > > > > >> 20x1 webinars > >> > > > > >> > > > > > > > > >> > > > > >> - This can be reproduced > stable > >> and > >> > > > > >> > repetitively > >> > > > > >> > > > > > > > > >> > > > > >> - The issue starts with API > >> calls > >> > > > taking > >> > > > > >> > 10sec++ > >> > > > > >> > > > and > >> > > > > >> > > > > > > > getting > >> > > > > >> > > > > > > > > >> more > >> > > > > >> > > > > > > > > >> > > > > slower. > >> > > > > >> > > > > > > > > >> > > > > >> Until the OpenMeetings Tomcat > >> > instance > >> > > > > >> crashes > >> > > > > >> > > > > > > > > >> > > > > >> - The issue also manifests > that > >> > > > -BEFORE- > >> > > > > >> the > >> > > > > >> > > > server > >> > > > > >> > > > > > > > crashes > >> > > > > >> > > > > > > > > >> you > >> > > > > >> > > > > > > > > >> > can > >> > > > > >> > > > > > > > > >> > > > see > >> > > > > >> > > > > > > > > >> > > > > >> video pods not completing the > >> > > > > >> initialisation in > >> > > > > >> > > the > >> > > > > >> > > > > > > > > conference > >> > > > > >> > > > > > > > > >> > room > >> > > > > >> > > > > > > > > >> > > > > >> itself. > >> > > > > >> > > > > > > > > >> > > > > >> For example missing video pods > or > >> > > video > >> > > > > pods > >> > > > > >> > > > without > >> > > > > >> > > > > a > >> > > > > >> > > > > > > > webcam > >> > > > > >> > > > > > > > > >> > > stream. > >> > > > > >> > > > > > > > > >> > > > > >> Likely to be linked to slow > >> running > >> > > API > >> > > > or > >> > > > > >> > > > web-socket > >> > > > > >> > > > > > > calls > >> > > > > >> > > > > > > > > >> > > > > >> => I can deliver data samples > or > >> > > > > >> screenshots if > >> > > > > >> > > > > > required > >> > > > > >> > > > > > > > via > >> > > > > >> > > > > > > > > >> our > >> > > > > >> > > > > > > > > >> > > > > >> confluence > >> > > > > >> > > > > > > > > >> > > > > >> space. > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> *Hardware and software:* > >> > > > > >> > > > > > > > > >> > > > > >> - Server and OpenMeetings > >> Instance > >> > is > >> > > > > >> isolated > >> > > > > >> > > on > >> > > > > >> > > > a > >> > > > > >> > > > > > > > > separated > >> > > > > >> > > > > > > > > >> > > > hardware > >> > > > > >> > > > > > > > > >> > > > > >> and > >> > > > > >> > > > > > > > > >> > > > > >> has 2GB of memory allocated > >> > > > > >> > > > > > > > > >> > > > > >> - There is no spike on KMS or > >> > > Database > >> > > > > >> > > > > > > > hardware/CPU/memory. > >> > > > > >> > > > > > > > > >> The > >> > > > > >> > > > > > > > > >> > > spike > >> > > > > >> > > > > > > > > >> > > > > is > >> > > > > >> > > > > > > > > >> > > > > >> only in the OpenMeetings Tomcat > >> > Server > >> > > > > >> instance > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> *Possible ways to mitigate > >> without > >> > > code > >> > > > > >> > changes:* > >> > > > > >> > > > > > > > > >> > > > > >> - You can mitigate part of > this > >> > issue > >> > > > if > >> > > > > >> you > >> > > > > >> > > > spread > >> > > > > >> > > > > > the > >> > > > > >> > > > > > > > > users > >> > > > > >> > > > > > > > > >> to > >> > > > > >> > > > > > > > > >> > > > enter > >> > > > > >> > > > > > > > > >> > > > > >> over a longer time period. > >> However > >> > it > >> > > > > needs > >> > > > > >> > more > >> > > > > >> > > > than > >> > > > > >> > > > > > > 10min > >> > > > > >> > > > > > > > > >> > > separation > >> > > > > >> > > > > > > > > >> > > > > to > >> > > > > >> > > > > > > > > >> > > > > >> enter without issues for 50-60 > >> > > > > participants > >> > > > > >> > > > > > > > > >> > > > > >> - You can mitigate part of > this > >> > issue > >> > > > if > >> > > > > >> you > >> > > > > >> > for > >> > > > > >> > > > > > example > >> > > > > >> > > > > > > > > >> create > >> > > > > >> > > > > > > > > >> > the > >> > > > > >> > > > > > > > > >> > > > > >> room-hash in a different > process > >> > (like > >> > > > 1h > >> > > > > >> > before > >> > > > > >> > > > > using) > >> > > > > >> > > > > > > and > >> > > > > >> > > > > > > > > >> once > >> > > > > >> > > > > > > > > >> > all > >> > > > > >> > > > > > > > > >> > > > > >> hashes > >> > > > > >> > > > > > > > > >> > > > > >> are created you enter the > >> conference > >> > > > room. > >> > > > > >> It > >> > > > > >> > > still > >> > > > > >> > > > > > leads > >> > > > > >> > > > > > > > to > >> > > > > >> > > > > > > > > >> > issues, > >> > > > > >> > > > > > > > > >> > > > but > >> > > > > >> > > > > > > > > >> > > > > >> you can enter up to 100 users > >> within > >> > > > > >> 5-10min, > >> > > > > >> > if > >> > > > > >> > > > you > >> > > > > >> > > > > > just > >> > > > > >> > > > > > > > use > >> > > > > >> > > > > > > > > >> the > >> > > > > >> > > > > > > > > >> > > > links, > >> > > > > >> > > > > > > > > >> > > > > >> rather than create the link AND > >> > > entering > >> > > > > >> with > >> > > > > >> > the > >> > > > > >> > > > > link > >> > > > > >> > > > > > at > >> > > > > >> > > > > > > > the > >> > > > > >> > > > > > > > > >> same > >> > > > > >> > > > > > > > > >> > > > > >> time/process > >> > > > > >> > > > > > > > > >> > > > > >> - Increasing Tomcat to more > than > >> > 2GB > >> > > of > >> > > > > >> memory > >> > > > > >> > > per > >> > > > > >> > > > > > > Tomcat > >> > > > > >> > > > > > > > > >> > instance > >> > > > > >> > > > > > > > > >> > > > may > >> > > > > >> > > > > > > > > >> > > > > >> help, not sure by how much > though > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> => I think we should spend > >> further > >> > > time > >> > > > > and > >> > > > > >> > > > propose > >> > > > > >> > > > > > ways > >> > > > > >> > > > > > > > to > >> > > > > >> > > > > > > > > >> get > >> > > > > >> > > > > > > > > >> > rid > >> > > > > >> > > > > > > > > >> > > > of > >> > > > > >> > > > > > > > > >> > > > > >> those spikes. The mitigations > are > >> > not > >> > > > > >> realistic > >> > > > > >> > > to > >> > > > > >> > > > > > really > >> > > > > >> > > > > > > > be > >> > > > > >> > > > > > > > > >> able > >> > > > > >> > > > > > > > > >> > to > >> > > > > >> > > > > > > > > >> > > > use > >> > > > > >> > > > > > > > > >> > > > > >> in > >> > > > > >> > > > > > > > > >> > > > > >> practise. > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> *My proposal is:* > >> > > > > >> > > > > > > > > >> > > > > >> There is further analysis > needed: > >> > > > > >> > > > > > > > > >> > > > > >> - Capture all OpenMeetings > calls > >> > that > >> > > > > >> happen > >> > > > > >> > > > during > >> > > > > >> > > > > > the > >> > > > > >> > > > > > > > > create > >> > > > > >> > > > > > > > > >> > room > >> > > > > >> > > > > > > > > >> > > > > hash > >> > > > > >> > > > > > > > > >> > > > > >> and conference room-enter > >> > > > > >> > > > > > > > > >> > > > > >> - Measure call lengths and any > >> > calls > >> > > > > during > >> > > > > >> > the > >> > > > > >> > > > > create > >> > > > > >> > > > > > > > room > >> > > > > >> > > > > > > > > >> hash > >> > > > > >> > > > > > > > > >> > > and > >> > > > > >> > > > > > > > > >> > > > > >> conference room-enter and > >> specific > >> > CPU > >> > > > > >> spikes > >> > > > > >> > or > >> > > > > >> > > > > memory > >> > > > > >> > > > > > > > usage > >> > > > > >> > > > > > > > > >> > based > >> > > > > >> > > > > > > > > >> > > > on a > >> > > > > >> > > > > > > > > >> > > > > >> per call basis > >> > > > > >> > > > > > > > > >> > > > > >> - Eventually get a stack trace > >> or > >> > > have > >> > > > a > >> > > > > >> > profile > >> > > > > >> > > > > > > available > >> > > > > >> > > > > > > > > >> that > >> > > > > >> > > > > > > > > >> > > > exports > >> > > > > >> > > > > > > > > >> > > > > >> the current in memory objects > to > >> > > review > >> > > > > >> where > >> > > > > >> > and > >> > > > > >> > > > > what > >> > > > > >> > > > > > > > create > >> > > > > >> > > > > > > > > >> > those > >> > > > > >> > > > > > > > > >> > > > > spikes > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> Once a per-call analysis is > >> there it > >> > > > > should > >> > > > > >> be > >> > > > > >> > a > >> > > > > >> > > > lot > >> > > > > >> > > > > > more > >> > > > > >> > > > > > > > > easy > >> > > > > >> > > > > > > > > >> to > >> > > > > >> > > > > > > > > >> > > > > pinpoint > >> > > > > >> > > > > > > > > >> > > > > >> specific issues and propose > >> > > > improvements. > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> As with all performance > >> optimisation > >> > > > this > >> > > > > is > >> > > > > >> > > likely > >> > > > > >> > > > > to > >> > > > > >> > > > > > > need > >> > > > > >> > > > > > > > > >> more > >> > > > > >> > > > > > > > > >> > > > > >> discussion > >> > > > > >> > > > > > > > > >> > > > > >> once more detailed data is > >> > available. > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> Thanks, > >> > > > > >> > > > > > > > > >> > > > > >> Sebastian > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> Sebastian Wagner > >> > > > > >> > > > > > > > > >> > > > > >> Director Arrakeen Solutions, > >> > > > > OM-Hosting.com > >> > > > > >> > > > > > > > > >> > > > > >> > http://arrakeen-solutions.co.nz/ > >> > > > > >> > > > > > > > > >> > > > > >> https://om-hosting.com - > Cloud & > >> > > Server > >> > > > > >> > Hosting > >> > > > > >> > > > for > >> > > > > >> > > > > > > HTML5 > >> > > > > >> > > > > > > > > >> > > > > >> Video-Conferencing OpenMeetings > >> > > > > >> > > > > > > > > >> > > > > >> < > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > >> > > > > >> > > > > > > > > >> > > > > >> > > >> > > > > >> > > > > > > > > >> > > > > >> < > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > >> > > > > >> > > > > > > > > >> > > > > >> > > >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > -- > >> > > > > >> > > > > > > > > >> > > > > > Best regards, > >> > > > > >> > > > > > > > > >> > > > > > Maxim > >> > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > -- > >> > > > > >> > > > > > > > > >> > > > > Best regards, > >> > > > > >> > > > > > > > > >> > > > > Maxim > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > -- > >> > > > > >> > > > > > > > > >> > > Best regards, > >> > > > > >> > > > > > > > > >> > > Maxim > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> -- > >> > > > > >> > > > > > > > > >> Best regards, > >> > > > > >> > > > > > > > > >> Maxim > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > -- > >> > > > > >> > > > > > > > Best regards, > >> > > > > >> > > > > > > > Maxim > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > -- > >> > > > > >> > > > > > Best regards, > >> > > > > >> > > > > > Maxim > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > > >> > > > Best regards, > >> > > > > >> > > > Maxim > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > -- > >> > > > > >> > Best regards, > >> > > > > >> > Maxim > >> > > > > >> > > >> > > > > >> > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Best regards, > >> > > > > > Maxim > >> > > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Best regards, > >> > > > > Maxim > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best regards, > >> > > Maxim > >> > > > >> > > >> > >> > >> -- > >> Best regards, > >> Maxim > >> > > -- > Sebastian Wagner > Director Arrakeen Solutions, OM-Hosting.com > http://arrakeen-solutions.co.nz/ > https://om-hosting.com - Cloud & Server Hosting for HTML5 > Video-Conferencing OpenMeetings > < > https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url > > > < > https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url > > > -- Best regards, Maxim
