Re: neither CPU nor IO bound, but throttled performance

[Gunther Schadow]

Thank you Magnus. 68% steal. Indeed. You probably hit the target. Yes.

That explains the discrepancy. I need to watch and understand that CPU 
credits issue.


regards,
-Gunther

On 2/21/2019 4:08, Magnus Hagander wrote:
On Thu, Feb 21, 2019 at 12:34 AM Gunther > wrote:


Hi, I have an Amazon Linux based Postgresql 11 server here on a
t2.medium EC2 instance.

It is serving 24 worker processes that read jobs from a queue
(thanks to SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of
which are reading and writing business data to the database,
others are only reading, and some don't hit the business data at
all, only the queue.

Everything flows quite nicely. Except, I don't understand why I
can't max out the CPU or the IO, instead, IO is almost negligible
yet the CPU is at 30% hardly hitting 50%.

Here I give you a view of top:

top - 23:17:09 up 45 days, 2:07, 4 users, load average: 20.32,
18.92, 13.80 Tasks: 338 total, 24 running, 111 sleeping, 0
stopped, 0 zombie %Cpu(s): 28.7 us, 2.5 sy, 0.0 ni, 0.0 id, 0.0
wa, 0.0 hi, 0.0 si, 68.7 st


If I read that right, it's about 70% "steal". The description for this 
is "the percentage of time spent in involuntary wait by the virtual 
CPU or CPUs while the hypervisor was servicing another virtual 
processor.".


So basically, the CPU is spent dealing with other peoples VMs on the 
same hardware. Welcome to the multi-tenant cloud.


In particular, I believe T series instances get a limited number of 
CPU "credits" per hours. My guess is you've hit this limit and are 
thus being throttled. T series are not intended for persistent 
workloads. Either way, this is probably a question better asked at the 
Amazon EC2 forums rather than PostgreSQL as you'll find more people 
who know the EC2 interactions there.

--
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: neither CPU nor IO bound, but throttled performance

[Magnus Hagander]

On Thu, Feb 21, 2019 at 12:34 AM Gunther  wrote:

> Hi, I have an Amazon Linux based Postgresql 11 server here on a t2.medium
> EC2 instance.
>
> It is serving 24 worker processes that read jobs from a queue (thanks to
> SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of which are reading
> and writing business data to the database, others are only reading, and
> some don't hit the business data at all, only the queue.
>
> Everything flows quite nicely. Except, I don't understand why I can't max
> out the CPU or the IO, instead, IO is almost negligible yet the CPU is at
> 30% hardly hitting 50%.
>
> Here I give you a view of top:
>
> top - 23:17:09 up 45 days,  2:07,  4 users,  load average: 20.32, 18.92, 13.80
> Tasks: 338 total,  24 running, 111 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 28.7 us,  2.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 68.7 
> st
>
>
If I read that right, it's about 70% "steal". The description for this is
"the percentage of time spent in involuntary wait by the virtual CPU or
CPUs while the hypervisor was servicing another virtual processor.".

So basically, the CPU is spent dealing with other peoples VMs on the same
hardware. Welcome to the multi-tenant cloud.

In particular, I believe T series instances get a limited number of CPU
"credits" per hours. My guess is you've hit this limit and are thus being
throttled. T series are not intended for persistent workloads. Either way,
this is probably a question better asked at the Amazon EC2 forums rather
than PostgreSQL as you'll find more people who know the EC2 interactions
there.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: neither CPU nor IO bound, but throttled performance

[Justin Pryzby]

On Wed, Feb 20, 2019 at 06:32:49PM -0500, Gunther wrote:
> Hi, I have an Amazon Linux based Postgresql 11 server here on a t2.medium
> EC2 instance.
> 
> Everything flows quite nicely. Except, I don't understand why I can't max
> out the CPU or the IO, instead, IO is almost negligible yet the CPU is at
> 30% hardly hitting 50%.

> avg-cpu: %user %nice %system %iowait %steal %idle 36.41 0.00 3.80 0.00 59.78
> 0.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await
> r_await w_await svctm %util xvda 0.00 0.00 0.00 7.69 0.00 92.31 24.00 0.02

This is unreadable, please try to attach it ?

> previously I had hit 100 %util here, that was when I didn't have the tables
> so spread out over so many tablespaces. Now I have it spread out like in the
> olden days where you spread out your tables over many "spindles", and I did
> this here so I could see which tables or indexes would be bottlenecks.

What was the old storage configuration and what is it now ?

> So how can it be that queries take quite long without the process running at
> higher CPU%?

You said everything flows nicely, but take a long time, and "throttled", can
you show an high-level performance change ?  Other than %cpu or %io.

Justin