Re: [GENERAL] BDR, wal sender, high system cpu, mutex_lock_common

2017-10-11 Thread Craig Ringer 
On 12 October 2017 at 11:03, Andres Freund  wrote:
> On 2017-10-12 10:25:43 +0800, Craig Ringer wrote:
>> On 4 October 2017 at 00:21, milist ujang  wrote:
>> > On Tue, Oct 3, 2017 at 8:49 PM, Craig Ringer  wrote:
>> >>
>> >>
>> >> Can you get stacks please?
>> >>
>> >> Use -g
>> >
>> >
>> > # Events: 2K cpu-clock
>> > #
>> > # Overhead   Command  Shared ObjectSymbol
>> > #     .  
>> > #
>> > 86.96%  postgres  [kernel.kallsyms]  [k] __mutex_lock_common.isra.5
>> > |
>> > --- __mutex_lock_common.isra.5
>> > read
>>
>>
>> Unfortunately it looks like you're using a postgres built with
>> -fomit-frame-pointers (the default) on x64, with an older perf not
>> built with libunwind. This produces useless stacks.
>
> Just read this mail, but for libunwind to work you'd have to specify
> "--call-graph dwarf", no?

I think you're right. But only on a version of perf where it's
available and used.

I haven't recently checked if perf has finally grown the ability to
load external debug symbols either. It never used to.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] BDR, wal sender, high system cpu, mutex_lock_common

2017-10-11 Thread Andres Freund 
On 2017-10-12 10:25:43 +0800, Craig Ringer wrote:
> On 4 October 2017 at 00:21, milist ujang  wrote:
> > On Tue, Oct 3, 2017 at 8:49 PM, Craig Ringer  wrote:
> >>
> >>
> >> Can you get stacks please?
> >>
> >> Use -g
> >
> >
> > # Events: 2K cpu-clock
> > #
> > # Overhead   Command  Shared ObjectSymbol
> > #     .  
> > #
> > 86.96%  postgres  [kernel.kallsyms]  [k] __mutex_lock_common.isra.5
> > |
> > --- __mutex_lock_common.isra.5
> > read
> 
> 
> Unfortunately it looks like you're using a postgres built with
> -fomit-frame-pointers (the default) on x64, with an older perf not
> built with libunwind. This produces useless stacks.

Just read this mail, but for libunwind to work you'd have to specify
"--call-graph dwarf", no?

- Andres


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] BDR, wal sender, high system cpu, mutex_lock_common

2017-10-11 Thread Craig Ringer 
On 4 October 2017 at 00:21, milist ujang  wrote:
> On Tue, Oct 3, 2017 at 8:49 PM, Craig Ringer  wrote:
>>
>>
>> Can you get stacks please?
>>
>> Use -g
>
>
> # Events: 2K cpu-clock
> #
> # Overhead   Command  Shared ObjectSymbol
> #     .  
> #
> 86.96%  postgres  [kernel.kallsyms]  [k] __mutex_lock_common.isra.5
> |
> --- __mutex_lock_common.isra.5
> read


Unfortunately it looks like you're using a postgres built with
-fomit-frame-pointers (the default) on x64, with an older perf not
built with libunwind. This produces useless stacks.

You may need to recompile with -fno-omit-frame-pointer


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] BDR, wal sender, high system cpu, mutex_lock_common

2017-10-03 Thread milist ujang 
Hi Craig,

Anyway, this OS is guess OS in vmware (vsphere).
Thank for your response and help.

On Tue, Oct 3, 2017 at 8:49 PM, Craig Ringer  wrote:
>
>
> Can you get stacks please?
>
> Use -g
>

# Events: 2K cpu-clock
#
# Overhead   Command  Shared ObjectSymbol
#     .  
#
86.96%  postgres  [kernel.kallsyms]  [k] __mutex_lock_common.isra.5
|
--- __mutex_lock_common.isra.5
read

 2.85%  postgres  [kernel.kallsyms]  [k] do_raw_spin_lock
|
--- do_raw_spin_lock
   |
   |--90.48%-- read
   |
   |--8.33%-- recv
   |
--1.19%-- write

 2.44%  postgres  [kernel.kallsyms]  [k] mutex_unlock
|
--- mutex_unlock
read

 2.03%  postgres  [kernel.kallsyms]  [k] arch_local_irq_restore
|
--- arch_local_irq_restore
read

 1.32%  postgres  postgres   [.] ValidXLogRecord
|
--- ValidXLogRecord

 1.25%  postgres  [kernel.kallsyms]  [k] mutex_lock
|
--- mutex_lock
read


-- 
regards

ujang jaenudin | DBA Consultant (Freelancer)
http://ora62.wordpress.com
http://id.linkedin.com/pub/ujang-jaenudin/12/64/bab

Re: [GENERAL] BDR, wal sender, high system cpu, mutex_lock_common

2017-10-03 Thread Craig Ringer 
On 3 October 2017 at 19:45, milist ujang  wrote:

> Hi all,
>
> I've an environment 9.4 + bdr:
> PostgreSQL 9.4.4
>

You're on a pretty old  postgres-bdr. Update. You're missing a lot of fixes
from mainline.


> This is consolidation databases, in this machine there are around 250+ wal
> sender processes.
>

Not a great use case for BDR.

Consider pglogical.


>
> finally get which processes (wal senders) that are using mutexes:
>
> perf top -e task-clock -p 55382
>
>
Can you get stacks please?

Use -g

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services