Sry my test numbers are slightly off. It’s not 50-60 users, but 80. Exactly 80: 10 conference rooms 4x4 + 2 webinar rooms with 20 users each = 80 users
I update the cwiki page. I will create a branch to try out adding some performance metrics. To see how hard but also how useful they will be. I make a Jira for the investigation. Thanks Seb On Tue, 2 Feb 2021 at 11:38 PM, [email protected] <[email protected]> wrote: > well we need something else (not sure yet how useful it will be) > => It means you spend less time on running tests for days & weeks and > re-running them in 100 different configurations in a try-and-error approach > without knowing what to look at. > > I can point to the demo server, yeah. Not sure if that will do much. > > Thanks > Seb > > Sebastian Wagner > Director Arrakeen Solutions, OM-Hosting.com > http://arrakeen-solutions.co.nz/ > https://om-hosting.com - Cloud & Server Hosting for HTML5 > Video-Conferencing OpenMeetings > > <https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url> > <https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url> > > > On Tue, 2 Feb 2021 at 23:34, Maxim Solodovnik <[email protected]> > wrote: > >> well we need something else (not sure yet how useful it will be) >> >> BTW maybe we can use demo-next as server >> just to compare with your docker? >> >> On Tue, 2 Feb 2021 at 17:28, [email protected] <[email protected] >> > >> wrote: >> >> > Sorry but access log just counts calls. It doesn't contain call length. >> > Which is the thing interesting. >> > Also as you mentioned: We have a mix of HTTP and websocket calls. >> > Web-Socket doesn't show up in access logs. >> > >> > It's just not useful. You can find out some very basic metrics. But not >> > good enough for diagnosing. >> > >> > Thanks >> > Sebastian >> > >> > Sebastian Wagner >> > Director Arrakeen Solutions, OM-Hosting.com >> > http://arrakeen-solutions.co.nz/ >> > https://om-hosting.com - Cloud & Server Hosting for HTML5 >> > Video-Conferencing OpenMeetings >> > < >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > >> > < >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > >> > >> > >> > On Tue, 2 Feb 2021 at 23:16, Maxim Solodovnik <[email protected]> >> > wrote: >> > >> > > tomcat has accesslog valve >> > > it should be enabled by default >> > > >> > > On Tue, 2 Feb 2021 at 16:49, [email protected] < >> > [email protected]> >> > > wrote: >> > > >> > > > I'm not sure. Like I say: I stagger room entry by 5-10min. So it's >> not >> > > > really a DDoS surge of users. It's a steady but slow growth. >> > > > >> > > > You might be right, there could be such an issue somewhere. Just >> quite >> > > > difficult to find without call statistics rights now. >> > > > >> > > > Have we looked into enabling some basic performance logs? >> > > > If you just have all API/WebSocket invocations logged (configurable) >> > for >> > > > performance analytics, you can find those kind of issues very easy. >> > > > Capture them and sort by call length, call numbers, sort by top ten >> => >> > > And >> > > > you get very quickly to a result. >> > > > >> > > > Some of those performance logging frameworks are very easy to >> enable. >> > You >> > > > can just annotate methods in Java code. And depending on log >> settings >> > it >> > > > will then print those statistics to the log file. >> > > > Even for example into a format that can be further ingested into >> > > Prometheus >> > > > for performance monitoring and graphing of results. Or for example >> in >> > > case >> > > > of Prometheus generate a HTTP endpoint that exposes the metrics for >> > > > generating statistics. >> > > > >> > > > See: >> > > > >> > > > - https://github.com/prometheus/client_java >> > > > - >> > > > >> > > > >> > > >> > >> https://github.com/prometheus/client_java/blob/master/simpleclient_spring_web/src/main/java/io/prometheus/client/spring/web/PrometheusTimeMethod.java >> > > > - >> > > > >> > > > >> > > >> > >> https://prometheus.github.io/client_java/io/prometheus/client/spring/web/PrometheusTimeMethod.html >> > > > >> > > > There might be other alternatives to Prometheus. But it is the >> current >> > > tool >> > > > most widely supported and it seems with a lot of SDKs, examples and >> > > > support. If we would have such tools available now I think it would >> be >> > > > quite easy to pinpoint the bottlenecks. Doesn't need any JProfiler >> or >> > > > Yourkit. Those are useful but the setup is a bit harder and you >> > > constantly >> > > > end up enabling/disabling the profiling. >> > > > >> > > > Thanks, >> > > > Sebastian >> > > > >> > > > Sebastian Wagner >> > > > Director Arrakeen Solutions, OM-Hosting.com >> > > > http://arrakeen-solutions.co.nz/ >> > > > https://om-hosting.com - Cloud & Server Hosting for HTML5 >> > > > Video-Conferencing OpenMeetings >> > > > < >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > < >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > >> > > > >> > > > On Tue, 2 Feb 2021 at 22:31, Maxim Solodovnik <[email protected] >> > >> > > > wrote: >> > > > >> > > > > Previous time I saw such many-users-same-time issues >> > > > > it was because of too many Ajax requests in room >> > > > > I have moved lot's of them to WS messages and things get better >> > > > > >> > > > > Wicket Ajax requests blocks all pages, maybe further improvements >> are >> > > > > required >> > > > > >> > > > > >> > > > > On Tue, 2 Feb 2021 at 16:27, Maxim Solodovnik < >> [email protected]> >> > > > > wrote: >> > > > > >> > > > > > fair enough :) >> > > > > > >> > > > > > On Tue, 2 Feb 2021 at 16:26, [email protected] < >> > > > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > >> I think adding cores at some point will be good. >> > > > > >> But we need to get to some reasonable user numbers on a single >> > > > > >> core/reasonable memory. >> > > > > >> Once those numbers are good => Scale it up. >> > > > > >> >> > > > > >> I have a try with the threads and report back. >> > > > > >> >> > > > > >> Thanks, >> > > > > >> Seb >> > > > > >> >> > > > > >> Sebastian Wagner >> > > > > >> Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> http://arrakeen-solutions.co.nz/ >> > > > > >> https://om-hosting.com - Cloud & Server Hosting for HTML5 >> > > > > >> Video-Conferencing OpenMeetings >> > > > > >> < >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > >> > > > > >> < >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > >> > > > > >> >> > > > > >> >> > > > > >> On Tue, 2 Feb 2021 at 22:23, Maxim Solodovnik < >> > [email protected] >> > > > >> > > > > >> wrote: >> > > > > >> >> > > > > >> > OK >> > > > > >> > no cores if it is expensive >> > > > > >> > >> > > > > >> > just thought multithreaded application can benefit from >> multiple >> > > > cores >> > > > > >> :) >> > > > > >> > >> > > > > >> > On Tue, 2 Feb 2021 at 16:21, [email protected] < >> > > > > >> [email protected]> >> > > > > >> > wrote: >> > > > > >> > >> > > > > >> > > I don't really want to add more cores. The docker container >> > has >> > > > > >> exactly 1 >> > > > > >> > > core just for OpenMeetings. And 4GB memory. >> > > > > >> > > >> > > > > >> > > We can try with 2 cores. But the price tags on those >> > > improvements >> > > > > are >> > > > > >> > > getting into a range of not viable options. Except you >> improve >> > > the >> > > > > >> > > performance by a factor of 10. >> > > > > >> > > >> > > > > >> > > Thanks >> > > > > >> > > Seb >> > > > > >> > > >> > > > > >> > > Sebastian Wagner >> > > > > >> > > Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > https://om-hosting.com - Cloud & Server Hosting for HTML5 >> > > > > >> > > Video-Conferencing OpenMeetings >> > > > > >> > > < >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > >> > > > > >> > > < >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > On Tue, 2 Feb 2021 at 22:13, Maxim Solodovnik < >> > > > [email protected] >> > > > > > >> > > > > >> > > wrote: >> > > > > >> > > >> > > > > >> > > > Maybe you can add one more core to OM >> > > > > >> > > > how many do you have right now? >> > > > > >> > > > >> > > > > >> > > > On Tue, 2 Feb 2021 at 16:11, [email protected] < >> > > > > >> > > [email protected]> >> > > > > >> > > > wrote: >> > > > > >> > > > >> > > > > >> > > > > I will have a look with 300 and repeat it. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > BTW are you using dockerized OM? how are you passing >> `xmx` >> > > via >> > > > > >> > > > > CATALINA_OPTS >> > > > > >> > > > > ? >> > > > > >> > > > > => I have a custom Openmeetings docker container and I >> set >> > > > those >> > > > > >> via >> > > > > >> > > > > CATALINA_OPS that are passed into the OpenMeetings >> > instance. >> > > > > >> > > > > I can see in the cataline.out logs that it reads the >> > values >> > > in >> > > > > and >> > > > > >> > uses >> > > > > >> > > > it. >> > > > > >> > > > > >> > > > > >> > > > > Are you setting additional memory for docker? >> > > > > >> > > > > => The Docker container itself also has 4GB memory >> > > available. >> > > > > >> > > > > >> > > > > >> > > > > If you compare the graphs from the 2GB and 4GB test you >> > can >> > > > see >> > > > > >> that >> > > > > >> > > > memory >> > > > > >> > > > > usage in % has dropped by exactly 50%. So it seems >> pretty >> > > > > >> convincing >> > > > > >> > > that >> > > > > >> > > > > those settings are all correctly applied. >> > > > > >> > > > > >> > > > > >> > > > > Thanks >> > > > > >> > > > > Seb >> > > > > >> > > > > >> > > > > >> > > > > Sebastian Wagner >> > > > > >> > > > > Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> > > > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > https://om-hosting.com - Cloud & Server Hosting for >> HTML5 >> > > > > >> > > > > Video-Conferencing OpenMeetings >> > > > > >> > > > > < >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > >> > > > > >> > > > > < >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Tue, 2 Feb 2021 at 22:04, Maxim Solodovnik < >> > > > > >> [email protected]> >> > > > > >> > > > > wrote: >> > > > > >> > > > > >> > > > > >> > > > > > the default is 150 >> > > > > >> > > > > > could you set to 300? >> > > > > >> > > > > > we will see is there will be improvement >> > > > > >> > > > > > >> > > > > >> > > > > > BTW are you using dockerized OM? how are you passing >> > `xmx` >> > > > via >> > > > > >> > > > > > CATALINA_OPTS >> > > > > >> > > > > > ? >> > > > > >> > > > > > Are you setting additional memory for docker? >> > > > > >> > > > > > >> > > > > >> > > > > > On Tue, 2 Feb 2021 at 16:00, [email protected] < >> > > > > >> > > > > [email protected]> >> > > > > >> > > > > > wrote: >> > > > > >> > > > > > >> > > > > >> > > > > > > I can try and re-run, how many would you recommend >> > worth >> > > > > >> trying >> > > > > >> > for >> > > > > >> > > > > this >> > > > > >> > > > > > > scenario ? >> > > > > >> > > > > > > >> > > > > >> > > > > > > Thanks >> > > > > >> > > > > > > Seb >> > > > > >> > > > > > > >> > > > > >> > > > > > > Sebastian Wagner >> > > > > >> > > > > > > Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> > > > > > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > https://om-hosting.com - Cloud & Server Hosting >> for >> > > HTML5 >> > > > > >> > > > > > > Video-Conferencing OpenMeetings >> > > > > >> > > > > > > < >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > >> > > > > >> > > > > > > < >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > On Tue, 2 Feb 2021 at 21:56, Maxim Solodovnik < >> > > > > >> > > [email protected]> >> > > > > >> > > > > > > wrote: >> > > > > >> > > > > > > >> > > > > >> > > > > > > > Have you tried to increase maxThreads for Tomcat? >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > On Tue, 2 Feb 2021 at 15:26, >> [email protected] >> > < >> > > > > >> > > > > > > [email protected]> >> > > > > >> > > > > > > > wrote: >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > > I doubled it to 4GB OpenMeetings and 4GB KMS. I >> > > > updated >> > > > > >> the >> > > > > >> > > > docker >> > > > > >> > > > > > > > instance >> > > > > >> > > > > > > > > to run Openmeetings with xms=2GB and Xmx=4GB. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > And I did run exactly the same test again: >> > > > > >> > > > > > > > > - 50-60 users >> > > > > >> > > > > > > > > - staggered to enter in a time period around >> > > 5-10min >> > > > > >> > > > > > > > > - distributed into 10 conference rooms 4x4 >> and 2 >> > > > > webinars >> > > > > >> > with >> > > > > >> > > > 20 >> > > > > >> > > > > > > users >> > > > > >> > > > > > > > > each >> > > > > >> > > > > > > > > - each test runs calls the API to >> > > > login/createRoomHash >> > > > > >> and >> > > > > >> > > then >> > > > > >> > > > > load >> > > > > >> > > > > > > the >> > > > > >> > > > > > > > > URL with the room (plus start webcam/audio >> stream >> > in >> > > > the >> > > > > >> > > > conference >> > > > > >> > > > > > > > rooms) >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > The results look almost the same. There is >> hardly >> > > any >> > > > > >> > > > improvement: >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > - CPU still spikes to almost 100%, memory is >> > not >> > > a >> > > > > >> problem >> > > > > >> > > > > > > > > - Empty video pods as well as video pods >> where >> > > > webcam >> > > > > >> > stream >> > > > > >> > > > > > didn't >> > > > > >> > > > > > > > > start >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > There isn't a crash, but that is mostly >> because I >> > > > > stagger >> > > > > >> it >> > > > > >> > to >> > > > > >> > > > > enter >> > > > > >> > > > > > > the >> > > > > >> > > > > > > > > server over a 5-10min period. Which didn't >> crash >> > the >> > > > 2GB >> > > > > >> > > instance >> > > > > >> > > > > > > either. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > Comparison of the CPU graphs of both hardware >> > > > > >> configuration >> > > > > >> > and >> > > > > >> > > > > test >> > > > > >> > > > > > > > runs: >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > There is pretty much no improvement. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > There is some work on the application side >> needed. >> > > > This >> > > > > >> does >> > > > > >> > > not >> > > > > >> > > > > look >> > > > > >> > > > > > > > like >> > > > > >> > > > > > > > > getting better by throwing more hardware at it. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > It is really quite limiting to have no logs >> about >> > > any >> > > > > >> sort of >> > > > > >> > > > > > > performance >> > > > > >> > > > > > > > > indicators like call length to narrow down >> where >> > the >> > > > > >> > bottleneck >> > > > > >> > > > is. >> > > > > >> > > > > > > > > You may find some very low hanging fruits in >> terms >> > > of >> > > > > >> > > > optimisation >> > > > > >> > > > > if >> > > > > >> > > > > > > you >> > > > > >> > > > > > > > > can simply concentrate on the top ten calls and >> > > > optimise >> > > > > >> > those. >> > > > > >> > > > > > > > > Rather than looking at CPU and memory graphs. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > Thanks >> > > > > >> > > > > > > > > Sebastian >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > Sebastian Wagner >> > > > > >> > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> > > > > > > > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > > > https://om-hosting.com - Cloud & Server >> Hosting >> > for >> > > > > HTML5 >> > > > > >> > > > > > > > > Video-Conferencing OpenMeetings >> > > > > >> > > > > > > > > < >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > < >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > On Tue, 2 Feb 2021 at 17:18, >> > [email protected] >> > > < >> > > > > >> > > > > > > > [email protected]> >> > > > > >> > > > > > > > > wrote: >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > > Have we ever looked into which java method >> would >> > > > > require >> > > > > >> > the >> > > > > >> > > > most >> > > > > >> > > > > > > > > > resources/time during the process of entering >> > the >> > > > > >> > conference >> > > > > >> > > > > room ? >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Sebastian Wagner >> > > > > >> > > > > > > > > > Director Arrakeen Solutions, OM-Hosting.com >> > > > > >> > > > > > > > > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > > > > https://om-hosting.com - Cloud & Server >> Hosting >> > > for >> > > > > >> HTML5 >> > > > > >> > > > > > > > > > Video-Conferencing OpenMeetings >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > < >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > < >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > On Tue, 2 Feb 2021 at 16:48, Maxim >> Solodovnik < >> > > > > >> > > > > > [email protected]> >> > > > > >> > > > > > > > > > wrote: >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> While do load testing I did the following: >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> create Jmeter test loading "semistatic" >> > stateless >> > > > > error >> > > > > >> > page >> > > > > >> > > > > with >> > > > > >> > > > > > > 300 >> > > > > >> > > > > > > > > >> simultaneous threads (I can share this test >> it >> > is >> > > > > very >> > > > > >> > > simple) >> > > > > >> > > > > > > > > >> CPU usage of OM process was near to 100% >> > > > > >> > > > > > > > > >> the situation is better if Tomcat has more >> > > threads >> > > > > >> > > (maxThread >> > > > > >> > > > > > > > parameter) >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> I guess we need to check "The Ultimate >> Tomcat >> > > > > >> Performace >> > > > > >> > > > Guide" >> > > > > >> > > > > > :))) >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> On Tue, 2 Feb 2021 at 10:41, >> > > [email protected] >> > > > < >> > > > > >> > > > > > > > > [email protected] >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> wrote: >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > Also the spikes are on the CPU actually >> more >> > > than >> > > > > on >> > > > > >> the >> > > > > >> > > > > memory: >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing#PerformanceTesting-ClusterPerformancetestresult02-022021 >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > The spike is just 50-60 users. >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > Why would CPU spike to almost 100% just >> for >> > > that >> > > > > >> amount >> > > > > >> > of >> > > > > >> > > > > > users ? >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > I can try with 4GB for Openmeetings and >> > repeat >> > > > the >> > > > > >> test. >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > Thanks >> > > > > >> > > > > > > > > >> > Seb >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > Sebastian Wagner >> > > > > >> > > > > > > > > >> > Director Arrakeen Solutions, >> OM-Hosting.com >> > > > > >> > > > > > > > > >> > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > > > >> > https://om-hosting.com - Cloud & Server >> > > Hosting >> > > > > for >> > > > > >> > HTML5 >> > > > > >> > > > > > > > > >> > Video-Conferencing OpenMeetings >> > > > > >> > > > > > > > > >> > < >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > < >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > On Tue, 2 Feb 2021 at 16:34, Maxim >> > Solodovnik < >> > > > > >> > > > > > > [email protected] >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > wrote: >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > > On Tue, 2 Feb 2021 at 10:30, >> > > > > [email protected] >> > > > > >> < >> > > > > >> > > > > > > > > >> > [email protected]> >> > > > > >> > > > > > > > > >> > > wrote: >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > > I think what you mean is you have >> > > > OpenMeetings >> > > > > >> and >> > > > > >> > > MySQL >> > > > > >> > > > > and >> > > > > >> > > > > > > KMS >> > > > > >> > > > > > > > > on >> > > > > >> > > > > > > > > >> one >> > > > > >> > > > > > > > > >> > > > instance with 4GB. >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > But its 2GB Just for OpenMeetings. >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > I mean >> > > > > >> > > > > > > > > >> > > 4GB just for OM (demo-next) >> > > > > >> > > > > > > > > >> > > 8GB just for OM (demo-prod) >> > > > > >> > > > > > > > > >> > > and this might need to be increased in >> case >> > > of >> > > > > many >> > > > > >> > > users >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > Additionally Tomcat's maxThreads might >> need >> > > to >> > > > be >> > > > > >> > > > increased >> > > > > >> > > > > > > here: >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L74 >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > I suspect lot's of simultaneous users >> need >> > > more >> > > > > >> > > resources >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > KMS is separated with another 2GB >> > > > > >> > > > > > > > > >> > > > MySQL is on another server with >> another >> > 2GB >> > > > > >> > > > > > > > > >> > > > So that would be 6GB in total. But >> only 2 >> > > are >> > > > > >> > > allocated >> > > > > >> > > > to >> > > > > >> > > > > > > > > >> > OpenMeetings. >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > XmX=2GB for OpenMeetings should be >> enough >> > > and >> > > > > not >> > > > > >> > > crash >> > > > > >> > > > > with >> > > > > >> > > > > > > > 50-60 >> > > > > >> > > > > > > > > >> > users >> > > > > >> > > > > > > > > >> > > > entering the room at the same time. >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > Thanks >> > > > > >> > > > > > > > > >> > > > Sebastian >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > Sebastian Wagner >> > > > > >> > > > > > > > > >> > > > Director Arrakeen Solutions, >> > OM-Hosting.com >> > > > > >> > > > > > > > > >> > > > http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > > > >> > > > https://om-hosting.com - Cloud & >> Server >> > > > > Hosting >> > > > > >> for >> > > > > >> > > > HTML5 >> > > > > >> > > > > > > > > >> > > > Video-Conferencing OpenMeetings >> > > > > >> > > > > > > > > >> > > > < >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > < >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > On Tue, 2 Feb 2021 at 16:26, Maxim >> > > > Solodovnik < >> > > > > >> > > > > > > > > [email protected] >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> > > > wrote: >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > > Hello Sebastian, >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > It seems 2GB of RAM is not enough >> for >> > OM >> > > > > >> > > > > > > > > >> > > > > `OutOfMemoryError: Container >> > killed >> > > > due >> > > > > >> to >> > > > > >> > > > memory >> > > > > >> > > > > > > usage` >> > > > > >> > > > > > > > > >> > > > > I never use less than 4GB (8-16GB in >> > > > > >> production) >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > On Tue, 2 Feb 2021 at 09:54, Maxim >> > > > > Solodovnik < >> > > > > >> > > > > > > > > >> [email protected]> >> > > > > >> > > > > > > > > >> > > > > wrote: >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > On Tue, 2 Feb 2021 at 07:23, >> > > > > >> > > [email protected] >> > > > > >> > > > < >> > > > > >> > > > > > > > > >> > > > > [email protected]> >> > > > > >> > > > > > > > > >> > > > > > wrote: >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Hi, >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> I have been conducting a few more >> > > > > >> performance >> > > > > >> > and >> > > > > >> > > > > load >> > > > > >> > > > > > > > tests >> > > > > >> > > > > > > > > >> with >> > > > > >> > > > > > > > > >> > > the >> > > > > >> > > > > > > > > >> > > > > goal >> > > > > >> > > > > > > > > >> > > > > >> of increasing participants to >> 100++. >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> The challenge is: >> > > > > >> > > > > > > > > >> > > > > >> *If more then 50-60 users >> > dynamically >> > > > > >> create a >> > > > > >> > > room >> > > > > >> > > > > > Hash >> > > > > >> > > > > > > > > (using >> > > > > >> > > > > > > > > >> > > > > Soap/Rest >> > > > > >> > > > > > > > > >> > > > > >> API) and use that Hash to enter >> the >> > > > > >> conference >> > > > > >> > > room >> > > > > >> > > > > CPU >> > > > > >> > > > > > > and >> > > > > >> > > > > > > > > >> memory >> > > > > >> > > > > > > > > >> > > > > spikes >> > > > > >> > > > > > > > > >> > > > > >> and server crashes* >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > Can you share API call sequence? >> > > > > >> > > > > > > > > >> > > > > > Maybe we can write JMeter scenario >> > for >> > > > > this? >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > server crash is something bad >> > > > > >> > > > > > > > > >> > > > > > What is happening? Is it a JVM >> crash? >> > > Or >> > > > is >> > > > > >> the >> > > > > >> > > > system >> > > > > >> > > > > > low >> > > > > >> > > > > > > > of >> > > > > >> > > > > > > > > >> > > resources >> > > > > >> > > > > > > > > >> > > > > > and the kernel kills the >> > trouble-maker? >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> *Test scenario observations:* >> > > > > >> > > > > > > > > >> > > > > >> - It does not matter if those >> users >> > > try >> > > > > to >> > > > > >> > enter >> > > > > >> > > > the >> > > > > >> > > > > > > same >> > > > > >> > > > > > > > > >> room or >> > > > > >> > > > > > > > > >> > > > > >> separate >> > > > > >> > > > > > > > > >> > > > > >> rooms. In the above test scenario >> > > it's a >> > > > > >> mix of >> > > > > >> > > 4x4 >> > > > > >> > > > > > > > > conference >> > > > > >> > > > > > > > > >> > rooms >> > > > > >> > > > > > > > > >> > > > and >> > > > > >> > > > > > > > > >> > > > > >> 20x1 webinars >> > > > > >> > > > > > > > > >> > > > > >> - This can be reproduced stable >> and >> > > > > >> > repetitively >> > > > > >> > > > > > > > > >> > > > > >> - The issue starts with API >> calls >> > > > taking >> > > > > >> > 10sec++ >> > > > > >> > > > and >> > > > > >> > > > > > > > getting >> > > > > >> > > > > > > > > >> more >> > > > > >> > > > > > > > > >> > > > > slower. >> > > > > >> > > > > > > > > >> > > > > >> Until the OpenMeetings Tomcat >> > instance >> > > > > >> crashes >> > > > > >> > > > > > > > > >> > > > > >> - The issue also manifests that >> > > > -BEFORE- >> > > > > >> the >> > > > > >> > > > server >> > > > > >> > > > > > > > crashes >> > > > > >> > > > > > > > > >> you >> > > > > >> > > > > > > > > >> > can >> > > > > >> > > > > > > > > >> > > > see >> > > > > >> > > > > > > > > >> > > > > >> video pods not completing the >> > > > > >> initialisation in >> > > > > >> > > the >> > > > > >> > > > > > > > > conference >> > > > > >> > > > > > > > > >> > room >> > > > > >> > > > > > > > > >> > > > > >> itself. >> > > > > >> > > > > > > > > >> > > > > >> For example missing video pods or >> > > video >> > > > > pods >> > > > > >> > > > without >> > > > > >> > > > > a >> > > > > >> > > > > > > > webcam >> > > > > >> > > > > > > > > >> > > stream. >> > > > > >> > > > > > > > > >> > > > > >> Likely to be linked to slow >> running >> > > API >> > > > or >> > > > > >> > > > web-socket >> > > > > >> > > > > > > calls >> > > > > >> > > > > > > > > >> > > > > >> => I can deliver data samples or >> > > > > >> screenshots if >> > > > > >> > > > > > required >> > > > > >> > > > > > > > via >> > > > > >> > > > > > > > > >> our >> > > > > >> > > > > > > > > >> > > > > >> confluence >> > > > > >> > > > > > > > > >> > > > > >> space. >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> *Hardware and software:* >> > > > > >> > > > > > > > > >> > > > > >> - Server and OpenMeetings >> Instance >> > is >> > > > > >> isolated >> > > > > >> > > on >> > > > > >> > > > a >> > > > > >> > > > > > > > > separated >> > > > > >> > > > > > > > > >> > > > hardware >> > > > > >> > > > > > > > > >> > > > > >> and >> > > > > >> > > > > > > > > >> > > > > >> has 2GB of memory allocated >> > > > > >> > > > > > > > > >> > > > > >> - There is no spike on KMS or >> > > Database >> > > > > >> > > > > > > > hardware/CPU/memory. >> > > > > >> > > > > > > > > >> The >> > > > > >> > > > > > > > > >> > > spike >> > > > > >> > > > > > > > > >> > > > > is >> > > > > >> > > > > > > > > >> > > > > >> only in the OpenMeetings Tomcat >> > Server >> > > > > >> instance >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> *Possible ways to mitigate >> without >> > > code >> > > > > >> > changes:* >> > > > > >> > > > > > > > > >> > > > > >> - You can mitigate part of this >> > issue >> > > > if >> > > > > >> you >> > > > > >> > > > spread >> > > > > >> > > > > > the >> > > > > >> > > > > > > > > users >> > > > > >> > > > > > > > > >> to >> > > > > >> > > > > > > > > >> > > > enter >> > > > > >> > > > > > > > > >> > > > > >> over a longer time period. >> However >> > it >> > > > > needs >> > > > > >> > more >> > > > > >> > > > than >> > > > > >> > > > > > > 10min >> > > > > >> > > > > > > > > >> > > separation >> > > > > >> > > > > > > > > >> > > > > to >> > > > > >> > > > > > > > > >> > > > > >> enter without issues for 50-60 >> > > > > participants >> > > > > >> > > > > > > > > >> > > > > >> - You can mitigate part of this >> > issue >> > > > if >> > > > > >> you >> > > > > >> > for >> > > > > >> > > > > > example >> > > > > >> > > > > > > > > >> create >> > > > > >> > > > > > > > > >> > the >> > > > > >> > > > > > > > > >> > > > > >> room-hash in a different process >> > (like >> > > > 1h >> > > > > >> > before >> > > > > >> > > > > using) >> > > > > >> > > > > > > and >> > > > > >> > > > > > > > > >> once >> > > > > >> > > > > > > > > >> > all >> > > > > >> > > > > > > > > >> > > > > >> hashes >> > > > > >> > > > > > > > > >> > > > > >> are created you enter the >> conference >> > > > room. >> > > > > >> It >> > > > > >> > > still >> > > > > >> > > > > > leads >> > > > > >> > > > > > > > to >> > > > > >> > > > > > > > > >> > issues, >> > > > > >> > > > > > > > > >> > > > but >> > > > > >> > > > > > > > > >> > > > > >> you can enter up to 100 users >> within >> > > > > >> 5-10min, >> > > > > >> > if >> > > > > >> > > > you >> > > > > >> > > > > > just >> > > > > >> > > > > > > > use >> > > > > >> > > > > > > > > >> the >> > > > > >> > > > > > > > > >> > > > links, >> > > > > >> > > > > > > > > >> > > > > >> rather than create the link AND >> > > entering >> > > > > >> with >> > > > > >> > the >> > > > > >> > > > > link >> > > > > >> > > > > > at >> > > > > >> > > > > > > > the >> > > > > >> > > > > > > > > >> same >> > > > > >> > > > > > > > > >> > > > > >> time/process >> > > > > >> > > > > > > > > >> > > > > >> - Increasing Tomcat to more than >> > 2GB >> > > of >> > > > > >> memory >> > > > > >> > > per >> > > > > >> > > > > > > Tomcat >> > > > > >> > > > > > > > > >> > instance >> > > > > >> > > > > > > > > >> > > > may >> > > > > >> > > > > > > > > >> > > > > >> help, not sure by how much though >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> => I think we should spend >> further >> > > time >> > > > > and >> > > > > >> > > > propose >> > > > > >> > > > > > ways >> > > > > >> > > > > > > > to >> > > > > >> > > > > > > > > >> get >> > > > > >> > > > > > > > > >> > rid >> > > > > >> > > > > > > > > >> > > > of >> > > > > >> > > > > > > > > >> > > > > >> those spikes. The mitigations are >> > not >> > > > > >> realistic >> > > > > >> > > to >> > > > > >> > > > > > really >> > > > > >> > > > > > > > be >> > > > > >> > > > > > > > > >> able >> > > > > >> > > > > > > > > >> > to >> > > > > >> > > > > > > > > >> > > > use >> > > > > >> > > > > > > > > >> > > > > >> in >> > > > > >> > > > > > > > > >> > > > > >> practise. >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> *My proposal is:* >> > > > > >> > > > > > > > > >> > > > > >> There is further analysis needed: >> > > > > >> > > > > > > > > >> > > > > >> - Capture all OpenMeetings calls >> > that >> > > > > >> happen >> > > > > >> > > > during >> > > > > >> > > > > > the >> > > > > >> > > > > > > > > create >> > > > > >> > > > > > > > > >> > room >> > > > > >> > > > > > > > > >> > > > > hash >> > > > > >> > > > > > > > > >> > > > > >> and conference room-enter >> > > > > >> > > > > > > > > >> > > > > >> - Measure call lengths and any >> > calls >> > > > > during >> > > > > >> > the >> > > > > >> > > > > create >> > > > > >> > > > > > > > room >> > > > > >> > > > > > > > > >> hash >> > > > > >> > > > > > > > > >> > > and >> > > > > >> > > > > > > > > >> > > > > >> conference room-enter and >> specific >> > CPU >> > > > > >> spikes >> > > > > >> > or >> > > > > >> > > > > memory >> > > > > >> > > > > > > > usage >> > > > > >> > > > > > > > > >> > based >> > > > > >> > > > > > > > > >> > > > on a >> > > > > >> > > > > > > > > >> > > > > >> per call basis >> > > > > >> > > > > > > > > >> > > > > >> - Eventually get a stack trace >> or >> > > have >> > > > a >> > > > > >> > profile >> > > > > >> > > > > > > available >> > > > > >> > > > > > > > > >> that >> > > > > >> > > > > > > > > >> > > > exports >> > > > > >> > > > > > > > > >> > > > > >> the current in memory objects to >> > > review >> > > > > >> where >> > > > > >> > and >> > > > > >> > > > > what >> > > > > >> > > > > > > > create >> > > > > >> > > > > > > > > >> > those >> > > > > >> > > > > > > > > >> > > > > spikes >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> Once a per-call analysis is >> there it >> > > > > should >> > > > > >> be >> > > > > >> > a >> > > > > >> > > > lot >> > > > > >> > > > > > more >> > > > > >> > > > > > > > > easy >> > > > > >> > > > > > > > > >> to >> > > > > >> > > > > > > > > >> > > > > pinpoint >> > > > > >> > > > > > > > > >> > > > > >> specific issues and propose >> > > > improvements. >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> As with all performance >> optimisation >> > > > this >> > > > > is >> > > > > >> > > likely >> > > > > >> > > > > to >> > > > > >> > > > > > > need >> > > > > >> > > > > > > > > >> more >> > > > > >> > > > > > > > > >> > > > > >> discussion >> > > > > >> > > > > > > > > >> > > > > >> once more detailed data is >> > available. >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> Thanks, >> > > > > >> > > > > > > > > >> > > > > >> Sebastian >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> Sebastian Wagner >> > > > > >> > > > > > > > > >> > > > > >> Director Arrakeen Solutions, >> > > > > OM-Hosting.com >> > > > > >> > > > > > > > > >> > > > > >> http://arrakeen-solutions.co.nz/ >> > > > > >> > > > > > > > > >> > > > > >> https://om-hosting.com - Cloud & >> > > Server >> > > > > >> > Hosting >> > > > > >> > > > for >> > > > > >> > > > > > > HTML5 >> > > > > >> > > > > > > > > >> > > > > >> Video-Conferencing OpenMeetings >> > > > > >> > > > > > > > > >> > > > > >> < >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> < >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url >> > > > > >> > > > > > > > > >> > > > > >> > >> > > > > >> > > > > > > > > >> > > > > >> >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > > -- >> > > > > >> > > > > > > > > >> > > > > > Best regards, >> > > > > >> > > > > > > > > >> > > > > > Maxim >> > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > > -- >> > > > > >> > > > > > > > > >> > > > > Best regards, >> > > > > >> > > > > > > > > >> > > > > Maxim >> > > > > >> > > > > > > > > >> > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > > -- >> > > > > >> > > > > > > > > >> > > Best regards, >> > > > > >> > > > > > > > > >> > > Maxim >> > > > > >> > > > > > > > > >> > > >> > > > > >> > > > > > > > > >> > >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > >> -- >> > > > > >> > > > > > > > > >> Best regards, >> > > > > >> > > > > > > > > >> Maxim >> > > > > >> > > > > > > > > >> >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > -- >> > > > > >> > > > > > > > Best regards, >> > > > > >> > > > > > > > Maxim >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > -- >> > > > > >> > > > > > Best regards, >> > > > > >> > > > > > Maxim >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > -- >> > > > > >> > > > Best regards, >> > > > > >> > > > Maxim >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> > >> > > > > >> > -- >> > > > > >> > Best regards, >> > > > > >> > Maxim >> > > > > >> > >> > > > > >> >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Best regards, >> > > > > > Maxim >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Best regards, >> > > > > Maxim >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Best regards, >> > > Maxim >> > > >> > >> >> >> -- >> Best regards, >> Maxim >> > -- Sebastian Wagner Director Arrakeen Solutions, OM-Hosting.com http://arrakeen-solutions.co.nz/ https://om-hosting.com - Cloud & Server Hosting for HTML5 Video-Conferencing OpenMeetings <https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url> <https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url>
