Re: Performance testing of Guacamole

[Mike Jumper]

On Sat, Mar 10, 2018 at 11:16 AM, amarjeetxc  wrote:

> RDP protocol has option to do video redirection, in which case the video is
> not rendered on the server side but is streamed to RDP client and is played
> by the client. do you think that can be supported?
>
>
RDP's video redirection is actually specific to Windows Media Player and
Microsoft's own video format.


> My major concern is the CPU hogging by guacd for a single RDP connection.
> Playing a video via RDP will certainly use more CPU cycles on the guacd
> host
> but consuming 100% of the CPU means other services on the same host can not
> get enough CPU time. and so I am concerned that if more users connect to
> same host and do RDP, would all get fair share of the CPU and if some CPU
> will be left for the host system or other necessary services on the system.
>
>
While it is true that an intensive task like re-encoding YouTube video will
reduce the cycles available, keep in mind that:

1) The kernel will be dynamically scheduling things to allocate a
reasonable amount of CPU time to each process.
2) guacd will be dynamically adjusting the framerate based on time spent
and time available for encoding, as well as other factors.

An extremely heavy connection will not have zero impact, but it will also
not starve out all other processes. Things will be balanced as best as
possible.

I am going to now automate 25 user connections using Chrome connecting via
> Guacamole to 25 Win 10 VMs running a simple app inside browser session with
> no moving component.
>
>
For this to be a valid test, you will need to ensure:

1) The 25 Windows VMs are on separate hardware from the Guacamole server.
2) You are using 25 separate copies of Chrome, on 25 separate and
independent machines.

- Mike

Re: Performance testing of Guacamole

[amarjeetxc]

RDP protocol has option to do video redirection, in which case the video is
not rendered on the server side but is streamed to RDP client and is played
by the client. do you think that can be supported?

My major concern is the CPU hogging by guacd for a single RDP connection.
Playing a video via RDP will certainly use more CPU cycles on the guacd host
but consuming 100% of the CPU means other services on the same host can not
get enough CPU time. and so I am concerned that if more users connect to
same host and do RDP, would all get fair share of the CPU and if some CPU
will be left for the host system or other necessary services on the system.

I am going to now automate 25 user connections using Chrome connecting via
Guacamole to 25 Win 10 VMs running a simple app inside browser session with
no moving component. 



--
Sent from: 
http://apache-guacamole-general-user-mailing-list.2363388.n4.nabble.com/

Re: Performance testing of Guacamole

[Mike Jumper]

On Sat, Mar 10, 2018 at 10:19 AM, Antony Awaida  wrote:

> Thanks for the prompt reply Mike: Follow up questions:
>
> On Sat, Mar 10, 2018 at 10:10 AM, Mike Jumper 
> wrote:
>
>> On Sat, Mar 10, 2018 at 9:44 AM, Antony Awaida 
>> wrote:
>>
>>> Hi Mike:
>>>
>>> Is there a way to tell Guac if this is a video stream - just pass it
>>> through to the client without going through the codec process?
>>>
>>>
>> Not exactly, but that wouldn't be a workable solution in the general
>> case. Even if it were possible to hook into the video stream within the
>> browser within the remote desktop, that video stream is in some arbitrary
>> codec supported by that browser, initially negotiated between the site
>> streaming the content and the browser itself. There is no guarantee that
>> the codec agreed upon by that browser will be supported by the browser of
>> the user on the Guacamole connection, let alone all users sharing that
>> connection via Guacamole's screen sharing capabilities.
>>
>
> AA> We already tell end users to use Chrome when using Guacamole. It would
> not be unreasonable to tell them that, when watching video, they should use
> Chrome in the remote desktop. Thoughts? will this work?
>
>

No:

1) It is not possible. More on this below.
2) If it were possible, it would not be a workable solution for the project
as a whole for the reasons cited above.


>> More efficient handling of video streams is still possible long term
>> through either or both of the following:
>>
>> 1) Adding hardware encoding support. While PNG may not lend itself well
>> to this, JPEG and WebP both have hardware encoders via GPU, and the
>> compression used by PNG (deflate) does have hardware implementations (via
>> dedicated cards). Offloading encoding of graphics to hardware would remove
>> the majority of processing requirements for things like watching YouTube.
>> Simply adding hardware encoding support for JPEG and WebP would be
>> low-hanging fruit here, given the ready availability of those encoders and
>> the fact that Guacamole will already automatically switch over to JPEG/WebP
>> when it heuristically detects video.
>>
>
> AA> this would be harder to control since users will watch any video they
> want including PNG.
>
>

The decision regarding whether to encode graphics received over RDP as
JPEG, WebP, or PNG is made by the optimizer within guacd, not the user.

By the time Guacamole receives the data for what appears to be video, it
has already been decoded by the application playing the video and turned
into raw graphical operations by the RDP server. The data available to
Guacamole as an RDP client is essentially equivalent to the data visible on
your screen, and Guacamole will not have access to the original video
stream any more than your eyeballs do.

- Mike

Re: Performance testing of Guacamole

[Antony Awaida]

Thanks for the prompt reply Mike: Follow up questions:

On Sat, Mar 10, 2018 at 10:10 AM, Mike Jumper 
wrote:

> On Sat, Mar 10, 2018 at 9:44 AM, Antony Awaida  wrote:
>
>> Hi Mike:
>>
>> Is there a way to tell Guac if this is a video stream - just pass it
>> through to the client without going through the codec process?
>>
>>
> Not exactly, but that wouldn't be a workable solution in the general case.
> Even if it were possible to hook into the video stream within the browser
> within the remote desktop, that video stream is in some arbitrary codec
> supported by that browser, initially negotiated between the site streaming
> the content and the browser itself. There is no guarantee that the codec
> agreed upon by that browser will be supported by the browser of the user on
> the Guacamole connection, let alone all users sharing that connection via
> Guacamole's screen sharing capabilities.
>

AA> We already tell end users to use Chrome when using Guacamole. It would
not be unreasonable to tell them that, when watching video, they should use
Chrome in the remote desktop. Thoughts? will this work?


>
> More efficient handling of video streams is still possible long term
> through either or both of the following:
>
> 1) Adding hardware encoding support. While PNG may not lend itself well to
> this, JPEG and WebP both have hardware encoders via GPU, and the
> compression used by PNG (deflate) does have hardware implementations (via
> dedicated cards). Offloading encoding of graphics to hardware would remove
> the majority of processing requirements for things like watching YouTube.
> Simply adding hardware encoding support for JPEG and WebP would be
> low-hanging fruit here, given the ready availability of those encoders and
> the fact that Guacamole will already automatically switch over to JPEG/WebP
> when it heuristically detects video.
>

AA> this would be harder to control since users will watch any video they
want including PNG.


>
> 2) Implementing a Windows driver that implements the Guacamole protocol,
> similar to the work-in-progress Guacamole driver for the X.Org X11 server.
> Regardless of whether hardware encoding is added, this would offload the
> actual encoding of graphics to the remote desktop. This would not be an
> easy task, but would probably be a good option to have.
>
> - Mike
>
>


-- 
Antony Awaida
CEO
www.apporto.com

ᐧ

Re: Performance testing of Guacamole

[Mike Jumper]

On Sat, Mar 10, 2018 at 9:44 AM, Antony Awaida  wrote:

> Hi Mike:
>
> Is there a way to tell Guac if this is a video stream - just pass it
> through to the client without going through the codec process?
>
>
Not exactly, but that wouldn't be a workable solution in the general case.
Even if it were possible to hook into the video stream within the browser
within the remote desktop, that video stream is in some arbitrary codec
supported by that browser, initially negotiated between the site streaming
the content and the browser itself. There is no guarantee that the codec
agreed upon by that browser will be supported by the browser of the user on
the Guacamole connection, let alone all users sharing that connection via
Guacamole's screen sharing capabilities.

More efficient handling of video streams is still possible long term
through either or both of the following:

1) Adding hardware encoding support. While PNG may not lend itself well to
this, JPEG and WebP both have hardware encoders via GPU, and the
compression used by PNG (deflate) does have hardware implementations (via
dedicated cards). Offloading encoding of graphics to hardware would remove
the majority of processing requirements for things like watching YouTube.
Simply adding hardware encoding support for JPEG and WebP would be
low-hanging fruit here, given the ready availability of those encoders and
the fact that Guacamole will already automatically switch over to JPEG/WebP
when it heuristically detects video.

2) Implementing a Windows driver that implements the Guacamole protocol,
similar to the work-in-progress Guacamole driver for the X.Org X11 server.
Regardless of whether hardware encoding is added, this would offload the
actual encoding of graphics to the remote desktop. This would not be an
easy task, but would probably be a good option to have.

- Mike

Re: Performance testing of Guacamole

[Antony Awaida]

Hi Mike:

Is there a way to tell Guac if this is a video stream - just pass it
through to the client without going through the codec process?

Thanks,
Antony


ᐧ

On Sat, Mar 10, 2018 at 9:17 AM, Mike Jumper 
wrote:

> On Mar 10, 2018 04:12, "amarjeetxc"  wrote:
>
> ...
> Play any You tube video on target VM in Google chrome/IE.
> This is single session through Guacamole.
>
> Findings:
> High, almost 100% CPU usage by “guacd” process on CentOS server.
> ...
>
> Please suggest the reason behind
>
>
> You are asking guac to re-encode a YouTube video in real-time.
>
> and any possible quick workaround.
>
>
> Honestly: rethink your testing methodology, and stop using a single
> connection watching YouTube as a performance test.
>
> You need a larger sample size, reasonable analogs for typical remote
> desktop use, and subjective (human being) measurements of responsiveness.
>
> See my original email.
>
> - Mike
>
>


-- 
Antony Awaida
CEO
www.apporto.com

Re: Performance testing of Guacamole

[Mike Jumper]

On Mar 10, 2018 04:12, "amarjeetxc"  wrote:

...
Play any You tube video on target VM in Google chrome/IE.
This is single session through Guacamole.

Findings:
High, almost 100% CPU usage by “guacd” process on CentOS server.
...

Please suggest the reason behind


You are asking guac to re-encode a YouTube video in real-time.

and any possible quick workaround.


Honestly: rethink your testing methodology, and stop using a single
connection watching YouTube as a performance test.

You need a larger sample size, reasonable analogs for typical remote
desktop use, and subjective (human being) measurements of responsiveness.

See my original email.

- Mike

Re: Performance testing of Guacamole

[amarjeetxc]

Hi Team, 

Environment:
Guacamole release 0.9.14  installed on CentOS 7.0 (final) server having 4
vCPUs.  ( I tried with the latest CODE from th eGIT as well. )
RDP session of Windows 10 VM through Guacamole.
Play any You tube video on target VM in Google chrome/IE.
This is single session through Guacamole.

Findings:
High, almost 100% CPU usage by “guacd” process on CentOS server.
guacd process contains 9 threads out of which only a SINGLE thread eats 100%
CPU and hence the process. (Image 1)
This is the thread which handles TCP connection with target VM over which
RDP data flows. 
Doing a strace on this process shows it is getting frequent timeouts in
select and frequent data on poll (Image 2).
Image 3 shows a gdb back trace of the same thread. Just to see call stack.
When the video stops, CPU usage falls back to 5%.

Please find the below screenshots.


 


 


 


 

Please suggest the reason behind and any possible quick workaround.






--
Sent from: 
http://apache-guacamole-general-user-mailing-list.2363388.n4.nabble.com/

Re: Performance testing of Guacamole

[ivanmarcus]

I see Mike's response to this, which was thorough and useful to know for 
my own planning.



That said, just to give you a comparison, I tried a similar test on a 
Ubuntu 16.04 Vbox VM (1gb RAM). With two concurrent RDP users, one 
streaming sometining on youtube, the load average was ~0.15, and guacd 
ranged from ~15% - 27% CPU...



On 7/03/2018 5:55 p.m., Amarjeet Singh wrote:

Hi Mike,

Now most of my feature related issues are known and most of them are 
fix, I moved to performance testing.


I am running Guacamole on a CentOS 7 VM, running on latest VMWare ESX 
on Dell 610. I did the test at night so there was no other load on the 
server.

My VM has 2 vCPU and 4 GB of RAM.

When I run a single session with Guacamole and open MSN and leave it 
open, I can see that the CPU utilization on my server fluctuates 
between 3% to 30%. Average is 5-6%, but sometimes it would hit 30% also.


Then I started a HD video in Youtube and left it. I see now that CPU 
on my server is almost 100% utilized.

The Video is not smooth anyway.
The process consuming all the CPU is guacd

With just a single user can guacd consume so much CPU?

Howe can we limit this.

I have to test this server to supprot around 50 simultaneous users. 
Although users are not going to do Youtube but with 5% of average CPU 
consumption for MSN.com for a single user, I dont see I could support 
more than 20 sessions.


What is your view and if you have recommendation to optimize CPU 
utilization by guacd.


Except for font smoothing, all other options on rdp are turned off. 
Bitmap caching and compression is ON.


Regards,
Amarjeet Singh

Re: Performance testing of Guacamole

[Mike Jumper]

On Tue, Mar 6, 2018 at 8:55 PM, Amarjeet Singh  wrote:

> Hi Mike,
>
>
When creating a new thread, please address the list as a whole. The idea
behind the mailing lists is to facilitate help/discussion within the entire
community. Calling out an individual at the beginning defeats that.

Now most of my feature related issues are known and most of them are fix, I
> moved to performance testing.
> ...
> When I run a single session with Guacamole and open MSN and leave it open,
> I can see that the CPU utilization on my server fluctuates between 3% to
> 30%. Average is 5-6%, but sometimes it would hit 30% also.
>
>
Instantaneous sampling of CPU usage is an extremely poor metric for gauging
overall scalability, particularly for something as subjective as remote
desktop performance. Load average is a better metric, but still not as good
as an actual load test. To perform such a load test, you would need to
actually connect multiple, independent machines (each with their own
browser instance) and use those machines to interact with separately-hosted
remote desktops, gauging subjective performance as load increases.

I actually have done exactly the above by:

1) Creating simulated but realistic load by scripting a remote desktop
session using image recognition software (Sikuli - http://www.sikuli.org/)
2) Automating the deployment of such simulated users by creating an image
in EC2
3) Gradually increasing the size of the simulated load from tens to hundreds
4) As load increases, relying on actual humans (who are connected to the
same guac server) to continuously use their remote desktops and report when
performance appears degraded

Based on these tests, we found that a typical server should be fine so long
as roughly 1 CPU core and 2 GB of memory are available for every 25
concurrent users at peak. Subjective performance of any particular
individual's remote desktop should not degrade until that level of overall
load is exceeded, and even then such degradation is gradual.

You should also be sure to modify Tomcat's server.xml to specify the "NIO"
connector. Some configurations use the blocking I/O connector by default,
which works fine but has issues scaling for large numbers of long-lived
connections like those typical of a Guacamole deployment.


> I have to test this server to supprot around 50 simultaneous users.
>

Based on the above load tests, you would need roughly 2 CPU cores and 4 GB
of memory to support that load at peak for normal remote desktop use. If
you are virtualizing things, this will also depend on how well-allocated
your server resources are.

Although users are not going to do Youtube but with 5% of average CPU
> consumption for MSN.com for a single user, I dont see I could support more
> than 20 sessions.
>
>
This is not how things work in practice. The CPU consumption of guacd is
very bursty, its built-in optimizer continuously tracks and adjusts for
response/processing times, and the OS kernel does a very good job of
scheduling tasks given load. Even assuming that 5% CPU usage were required
on average, the question is not "what is 100% divided by 5%" but "how much
less than 5% produces a subjective difference".

- Mike

Performance testing of Guacamole

[Amarjeet Singh]

Hi Mike,

Now most of my feature related issues are known and most of them are fix, I
moved to performance testing.

I am running Guacamole on a CentOS 7 VM, running on latest VMWare ESX on
Dell 610. I did the test at night so there was no other load on the server.
My VM has 2 vCPU and 4 GB of RAM.

When I run a single session with Guacamole and open MSN and leave it open,
I can see that the CPU utilization on my server fluctuates between 3% to
30%. Average is 5-6%, but sometimes it would hit 30% also.

Then I started a HD video in Youtube and left it. I see now that CPU on my
server is almost 100% utilized.
The Video is not smooth anyway.
The process consuming all the CPU is guacd

With just a single user can guacd consume so much CPU?

Howe can we limit this.

I have to test this server to supprot around 50 simultaneous users.
Although users are not going to do Youtube but with 5% of average CPU
consumption for MSN.com for a single user, I dont see I could support more
than 20 sessions.

What is your view and if you have recommendation to optimize CPU
utilization by guacd.

Except for font smoothing, all other options on rdp are turned off. Bitmap
caching and compression is ON.

Regards,
Amarjeet Singh