Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-11-23 Thread Tatsuo Ishii 
> On Wed, Nov 23, 2011 at 10:42 PM, Tatsuo Ishii  wrote:
>>> Not wanting to be impatient, but I'm very concerned about this
>>> problem, since its impossible to predict when it will occur.  Is there
>>> additional information that I can provide to investigate this further?
>>
>> I really need to know where pgpool is looping.
> 
> OK, how can I capture that information?

You already attached to the pgpool process. So just type "n" (for
"next") will tell you next line to execute. If pgpool really loops,
"n" should show the same line after some repeating "n".
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-11-23 Thread Lonni J Friedman 
On Wed, Nov 23, 2011 at 10:42 PM, Tatsuo Ishii  wrote:
>> Not wanting to be impatient, but I'm very concerned about this
>> problem, since its impossible to predict when it will occur.  Is there
>> additional information that I can provide to investigate this further?
>
> I really need to know where pgpool is looping.

OK, how can I capture that information?
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-11-23 Thread Tatsuo Ishii 
> Not wanting to be impatient, but I'm very concerned about this
> problem, since its impossible to predict when it will occur.  Is there
> additional information that I can provide to investigate this further?

I really need to know where pgpool is looping.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> thanks
> 
> On Tue, Nov 22, 2011 at 10:11 AM, Lonni J Friedman  wrote:
>> This hadn't reproduced in a long time, but we ugpraded to pgpool-3.1 a
>> week ago, and this morning I found a pgpool process that was consuming
>> 100% CPU, and had been running for a week (although wasn't consuming
>> 100% CPU the entire time).  Something else weird is that it showed an
>> active, idle connection from a client system which had only been up
>> for the past 21 hours.  Anyway, here's the backtrace from the process
>> (gdb hung at the very bottom):
>>
>> [root ~]# gdb pgpool 31293
>> GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later 
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> ...
>> Reading symbols from /usr/sbin/pgpool...Reading symbols from
>> /usr/lib/debug/usr/sbin/pgpool.debug...done.
>> done.
>> Attaching to program: /usr/sbin/pgpool, process 31293
>> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libpq.so.5
>> Reading symbols from /usr/lib64/libpcp.so.0...Reading symbols from
>> /usr/lib/debug/usr/lib64/libpcp.so.0.0.0.debug...done.
>> done.
>> Loaded symbols for /usr/lib64/libpcp.so.0
>> Reading symbols from /lib64/libpam.so.0...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libpam.so.0
>> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libssl.so.10
>> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libcrypto.so.10
>> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libcrypt.so.1
>> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libresolv.so.2
>> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libnsl.so.1
>> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libm.so.6
>> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libc.so.6
>> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libgssapi_krb5.so.2
>> Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
>> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
>> found)...done.
>> [Thread debugging using libthread_db enabled]
>> Loaded symbols for /lib64/libpthread.so.0
>> Reading symbols from /lib64/libaudit.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libaudit.so.1
>> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libdl.so.2
>> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libkrb5.so.3
>> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libcom_err.so.2
>> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libk5crypto.so.3
>> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libz.so.1
>> Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libfreebl3.so
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libkrb5support.so.0
>> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libkeyutils.so.1
>> Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/liblber-2.4.so.2
>> Reading symbols from /usr/lib64/libsasl2.so.2...

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-11-23 Thread Lonni J Friedman 
Not wanting to be impatient, but I'm very concerned about this
problem, since its impossible to predict when it will occur.  Is there
additional information that I can provide to investigate this further?

thanks

On Tue, Nov 22, 2011 at 10:11 AM, Lonni J Friedman  wrote:
> This hadn't reproduced in a long time, but we ugpraded to pgpool-3.1 a
> week ago, and this morning I found a pgpool process that was consuming
> 100% CPU, and had been running for a week (although wasn't consuming
> 100% CPU the entire time).  Something else weird is that it showed an
> active, idle connection from a client system which had only been up
> for the past 21 hours.  Anyway, here's the backtrace from the process
> (gdb hung at the very bottom):
>
> [root ~]# gdb pgpool 31293
> GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Reading symbols from /usr/sbin/pgpool...Reading symbols from
> /usr/lib/debug/usr/sbin/pgpool.debug...done.
> done.
> Attaching to program: /usr/sbin/pgpool, process 31293
> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libpq.so.5
> Reading symbols from /usr/lib64/libpcp.so.0...Reading symbols from
> /usr/lib/debug/usr/lib64/libpcp.so.0.0.0.debug...done.
> done.
> Loaded symbols for /usr/lib64/libpcp.so.0
> Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libpam.so.0
> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libssl.so.10
> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/libcrypto.so.10
> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/libgssapi_krb5.so.2
> Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libaudit.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libaudit.so.1
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libkrb5.so.3
> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libcom_err.so.2
> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libk5crypto.so.3
> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libfreebl3.so
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/libkrb5support.so.0
> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libkeyutils.so.1
> Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/liblber-2.4.so.2
> Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libsasl2.so.2
> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libselinux.so.1
> Reading symbols from /lib64/libnss_files.so.2...(no debug

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-11-22 Thread Lonni J Friedman 
This hadn't reproduced in a long time, but we ugpraded to pgpool-3.1 a
week ago, and this morning I found a pgpool process that was consuming
100% CPU, and had been running for a week (although wasn't consuming
100% CPU the entire time).  Something else weird is that it showed an
active, idle connection from a client system which had only been up
for the past 21 hours.  Anyway, here's the backtrace from the process
(gdb hung at the very bottom):

[root ~]# gdb pgpool 31293
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/pgpool...Reading symbols from
/usr/lib/debug/usr/sbin/pgpool.debug...done.
done.
Attaching to program: /usr/sbin/pgpool, process 31293
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/libpcp.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libpcp.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libpcp.so.0
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreebl3.so
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x004192c0 in pool_process_query (frontend=0x2dd8fd0,
backend=0x187d540,
reset_request=) at pool_process_query.c:379
379 if (!pool_read_buffer_is_empty(MASTER(backend)) 
||
pool_is_query_in_progress())
Missing separate debuginfos, use: debuginfo-install
audit-libs-2.0.4-1.el6.x86_64 cyrus-sasl-lib-2.1.23-8.el6.x86_64
glibc-2.12-1.7.el6.x86_64 keyutils-lib

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-20 Thread Lonni J Friedman 
Nevermind, I figured out what I was doing wrong.  Now I just need for
this to hang again.

On Tue, Sep 20, 2011 at 9:08 AM, Lonni J Friedman  wrote:
> I tried to do that, but pgpool refuses to start reporting:
> -bash: /usr/lib/debug/usr/sbin/pgpool.debug: bad ELF interpreter: No
> such file or directory
>
> I'm puzzled why it fails, as it was built on the same server where I
> built the (working) release build of pgpool.
>
> $ file /usr/lib/debug/usr/sbin/pgpool.debug
> /usr/lib/debug/usr/sbin/pgpool.debug: ELF 64-bit LSB executable,
> x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs),
> for GNU/Linux 2.6.18, not stripped
> $ file /usr/sbin/pgpool
> /usr/sbin/pgpool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
> dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
>
>
>
>
> On Mon, Sep 19, 2011 at 7:10 PM, Tatsuo Ishii  wrote:
>> It's really hard to find the cause of the problem from a stack trace
>> without symbol tables... Is it possible to reinstalll pgpool binary
>> with debug symbols?
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>>> This happened again.  I ran the gdb command that you requested,
>>> however it occurred to me that the output may not be all that useful
>>> since I'm not running a debug build of pgpool:
>>> ###
>>> # gdb pgpool 2343
>>> GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later 
>>> 
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>> For bug reporting instructions, please see:
>>> ...
>>> Reading symbols from /usr/sbin/pgpool...(no debugging symbols found)...done.
>>> Attaching to program: /usr/sbin/pgpool, process 2343
>>> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libpq.so.5
>>> Reading symbols from /usr/lib64/libpcp.so.0...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libpcp.so.0
>>> Reading symbols from /lib64/libpam.so.0...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libpam.so.0
>>> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libssl.so.10
>>> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/libcrypto.so.10
>>> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libcrypt.so.1
>>> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libresolv.so.2
>>> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libnsl.so.1
>>> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
>>> Loaded symbols for /lib64/libm.so.6
>>> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
>>> Loaded symbols for /lib64/libc.so.6
>>> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /lib64/libgssapi_krb5.so.2
>>> Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
>>> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
>>> found)...done.
>>> [Thread debugging using libthread_db enabled]
>>> Loaded symbols for /lib64/libpthread.so.0
>>> Reading symbols from /lib64/libaudit.so.1...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libaudit.so.1
>>> Reading symbols from /lib64/libdl.so.2...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libdl.so.2
>>> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
>>> found)...done.
>>> Loaded symbols for /lib64/libkrb5.so.3
>>> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libcom_err.so.2
>>> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libk5crypto.so.3
>>> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
>>> Loaded symbols for /lib64/libz.so.1
>>> Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libfreebl3.so
>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>> Reading symb

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-20 Thread Lonni J Friedman 
I tried to do that, but pgpool refuses to start reporting:
-bash: /usr/lib/debug/usr/sbin/pgpool.debug: bad ELF interpreter: No
such file or directory

I'm puzzled why it fails, as it was built on the same server where I
built the (working) release build of pgpool.

$ file /usr/lib/debug/usr/sbin/pgpool.debug
/usr/lib/debug/usr/sbin/pgpool.debug: ELF 64-bit LSB executable,
x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs),
for GNU/Linux 2.6.18, not stripped
$ file /usr/sbin/pgpool
/usr/sbin/pgpool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped




On Mon, Sep 19, 2011 at 7:10 PM, Tatsuo Ishii  wrote:
> It's really hard to find the cause of the problem from a stack trace
> without symbol tables... Is it possible to reinstalll pgpool binary
> with debug symbols?
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
>> This happened again.  I ran the gdb command that you requested,
>> however it occurred to me that the output may not be all that useful
>> since I'm not running a debug build of pgpool:
>> ###
>> # gdb pgpool 2343
>> GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later 
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> ...
>> Reading symbols from /usr/sbin/pgpool...(no debugging symbols found)...done.
>> Attaching to program: /usr/sbin/pgpool, process 2343
>> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libpq.so.5
>> Reading symbols from /usr/lib64/libpcp.so.0...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libpcp.so.0
>> Reading symbols from /lib64/libpam.so.0...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libpam.so.0
>> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libssl.so.10
>> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libcrypto.so.10
>> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libcrypt.so.1
>> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libresolv.so.2
>> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libnsl.so.1
>> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libm.so.6
>> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libc.so.6
>> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libgssapi_krb5.so.2
>> Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
>> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
>> found)...done.
>> [Thread debugging using libthread_db enabled]
>> Loaded symbols for /lib64/libpthread.so.0
>> Reading symbols from /lib64/libaudit.so.1...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libaudit.so.1
>> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libdl.so.2
>> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
>> found)...done.
>> Loaded symbols for /lib64/libkrb5.so.3
>> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libcom_err.so.2
>> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libk5crypto.so.3
>> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/libz.so.1
>> Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libfreebl3.so
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libkrb5support.so.0
>> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libkeyutils.so.1
>> Reading sym

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-19 Thread Tatsuo Ishii 
It's really hard to find the cause of the problem from a stack trace
without symbol tables... Is it possible to reinstalll pgpool binary
with debug symbols?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> This happened again.  I ran the gdb command that you requested,
> however it occurred to me that the output may not be all that useful
> since I'm not running a debug build of pgpool:
> ###
> # gdb pgpool 2343
> GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Reading symbols from /usr/sbin/pgpool...(no debugging symbols found)...done.
> Attaching to program: /usr/sbin/pgpool, process 2343
> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libpq.so.5
> Reading symbols from /usr/lib64/libpcp.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libpcp.so.0
> Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libpam.so.0
> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libssl.so.10
> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/libcrypto.so.10
> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/libgssapi_krb5.so.2
> Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libaudit.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libaudit.so.1
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libkrb5.so.3
> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libcom_err.so.2
> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libk5crypto.so.3
> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libfreebl3.so
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /lib64/libkrb5support.so.0
> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libkeyutils.so.1
> Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/liblber-2.4.so.2
> Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libsasl2.so.2
> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libselinux.so.1
> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> 0x0044790a in ?? ()
> Missing separate debuginfos, use: debuginfo-install 
> pgpool-II-3.0.4-1.el6.x86_64
> (gdb) bt
> #0  0x0044790

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-18 Thread Lonni J Friedman 
This happened again.  I ran the gdb command that you requested,
however it occurred to me that the output may not be all that useful
since I'm not running a debug build of pgpool:
###
# gdb pgpool 2343
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/pgpool...(no debugging symbols found)...done.
Attaching to program: /usr/sbin/pgpool, process 2343
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/libpcp.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpcp.so.0
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreebl3.so
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
0x0044790a in ?? ()
Missing separate debuginfos, use: debuginfo-install pgpool-II-3.0.4-1.el6.x86_64
(gdb) bt
#0  0x0044790a in ?? ()
#1  0x00414547 in ?? ()
#2  0x0041762e in ?? ()
#3  0x0040a4cd in ?? ()
#4  0x00405345 in ?? ()
#5  0x004068dc in ?? ()
#6  0x004076dc in ?? ()
#7  0x0031ae41ec5d in __libc_start_main () from /lib64/libc.so.6
#8  0x00403bf9 in ?? ()
#9  0x7fff0663cfc8 in ?? ()
#10 0x001c in ?? ()
#11 0x0004 in ?? ()
#12 0x7fff0663d90b in ?? ()
#13 0x7fff0663dfe6 in ?? ()
#14 0x7fff

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-14 Thread Lonni J Friedman 
Thanks for your reply.  I'll do this the next time this happens (which
will likely be within a few days based on history).

On Wed, Sep 14, 2011 at 3:57 PM, Tatsuo Ishii  wrote:
> Please use gdb. For example,
>
> become postgres user (or root user)
> gdb pgpool 29191
> bt
> cont
> bt
> cont
> :
> :
> :
>
> This will give us an idea where it's looping.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
>> This problem has returned yet again:
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 29191 postgres  20   0 80192  14m 1544 R 89.8  0.2  51:15.91 pgpool
>>
>> postgres 29191  3.4  0.1  80192 14728 ?        R    Sep13  51:40
>> pgpool: lfriedman nightly 10.31.96.84(61698) idle
>>
>>
>> I'd really appreciate some input on how to debug this.
>>
>>
>> On Fri, Sep 9, 2011 at 8:11 AM, Lonni J Friedman  wrote:
>>> No one else has experienced this or has suggestions how to debug it?
>>>
>>> On Wed, Sep 7, 2011 at 12:49 PM, Lonni J Friedman  
>>> wrote:
 Greetings,
 I'm running pgpool-3.0.4 on a Linux-x86_64 server serving as a load
 balancer for a three server postgresql-9.0.4 cluster (1 master, 2
 standby).  I'm seeing strange behavior where a single pgpool process
 seems to hang after some period of time, and then consume 100% of the
 CPU.  I've seen this behavior happen twice since last Friday (when
 pgpool was brought online in my production environment).  At the
 moment the current hung process looks like this in 'ps auxww' output:

 postgres 19838 98.7  0.0  68856  2904 ?        R    Sep06 1027:36
 pgpool: lfriedman nightly 10.31.45.20(58277) idle


 In top, I see:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 19838 postgres  20   0 68856 2904 1072 R 100.0  0.0   1027:29 pgpool


 When to connect to the process with strace, there is no output, so I'm
 guessing the process is stuck spinning somewhere:
 # strace -p 19838
 Process 19838 attached - interrupt to quit
 ...
 ^CProcess 19838 detached

 One thing that i'm certain of is that the client IP (10.31.45.20)
 associated with the hung process has rebooted at least once since that
 process was spawned.  So pgpool seems to be in some confused state, as
 the client definitely severed the connection already.  I checked the
 pgpool log and there are no explicit references to PID 19838.  I'm at
 a loss how to debug this further, but clearly something is wrong
 somewhere, and this isn't normal/expected behavior.
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-14 Thread Tatsuo Ishii 
Please use gdb. For example,

become postgres user (or root user)
gdb pgpool 29191
bt
cont
bt
cont
:
:
:

This will give us an idea where it's looping.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> This problem has returned yet again:
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 29191 postgres  20   0 80192  14m 1544 R 89.8  0.2  51:15.91 pgpool
> 
> postgres 29191  3.4  0.1  80192 14728 ?RSep13  51:40
> pgpool: lfriedman nightly 10.31.96.84(61698) idle
> 
> 
> I'd really appreciate some input on how to debug this.
> 
> 
> On Fri, Sep 9, 2011 at 8:11 AM, Lonni J Friedman  wrote:
>> No one else has experienced this or has suggestions how to debug it?
>>
>> On Wed, Sep 7, 2011 at 12:49 PM, Lonni J Friedman  wrote:
>>> Greetings,
>>> I'm running pgpool-3.0.4 on a Linux-x86_64 server serving as a load
>>> balancer for a three server postgresql-9.0.4 cluster (1 master, 2
>>> standby).  I'm seeing strange behavior where a single pgpool process
>>> seems to hang after some period of time, and then consume 100% of the
>>> CPU.  I've seen this behavior happen twice since last Friday (when
>>> pgpool was brought online in my production environment).  At the
>>> moment the current hung process looks like this in 'ps auxww' output:
>>>
>>> postgres 19838 98.7  0.0  68856  2904 ?        R    Sep06 1027:36
>>> pgpool: lfriedman nightly 10.31.45.20(58277) idle
>>>
>>>
>>> In top, I see:
>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 19838 postgres  20   0 68856 2904 1072 R 100.0  0.0   1027:29 pgpool
>>>
>>>
>>> When to connect to the process with strace, there is no output, so I'm
>>> guessing the process is stuck spinning somewhere:
>>> # strace -p 19838
>>> Process 19838 attached - interrupt to quit
>>> ...
>>> ^CProcess 19838 detached
>>>
>>> One thing that i'm certain of is that the client IP (10.31.45.20)
>>> associated with the hung process has rebooted at least once since that
>>> process was spawned.  So pgpool seems to be in some confused state, as
>>> the client definitely severed the connection already.  I checked the
>>> pgpool log and there are no explicit references to PID 19838.  I'm at
>>> a loss how to debug this further, but clearly something is wrong
>>> somewhere, and this isn't normal/expected behavior.
> ___
> Pgpool-general mailing list
> Pgpool-general@pgfoundry.org
> http://pgfoundry.org/mailman/listinfo/pgpool-general
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-14 Thread Lonni J Friedman 
This problem has returned yet again:
  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29191 postgres  20   0 80192  14m 1544 R 89.8  0.2  51:15.91 pgpool

postgres 29191  3.4  0.1  80192 14728 ?RSep13  51:40
pgpool: lfriedman nightly 10.31.96.84(61698) idle


I'd really appreciate some input on how to debug this.


On Fri, Sep 9, 2011 at 8:11 AM, Lonni J Friedman  wrote:
> No one else has experienced this or has suggestions how to debug it?
>
> On Wed, Sep 7, 2011 at 12:49 PM, Lonni J Friedman  wrote:
>> Greetings,
>> I'm running pgpool-3.0.4 on a Linux-x86_64 server serving as a load
>> balancer for a three server postgresql-9.0.4 cluster (1 master, 2
>> standby).  I'm seeing strange behavior where a single pgpool process
>> seems to hang after some period of time, and then consume 100% of the
>> CPU.  I've seen this behavior happen twice since last Friday (when
>> pgpool was brought online in my production environment).  At the
>> moment the current hung process looks like this in 'ps auxww' output:
>>
>> postgres 19838 98.7  0.0  68856  2904 ?        R    Sep06 1027:36
>> pgpool: lfriedman nightly 10.31.45.20(58277) idle
>>
>>
>> In top, I see:
>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 19838 postgres  20   0 68856 2904 1072 R 100.0  0.0   1027:29 pgpool
>>
>>
>> When to connect to the process with strace, there is no output, so I'm
>> guessing the process is stuck spinning somewhere:
>> # strace -p 19838
>> Process 19838 attached - interrupt to quit
>> ...
>> ^CProcess 19838 detached
>>
>> One thing that i'm certain of is that the client IP (10.31.45.20)
>> associated with the hung process has rebooted at least once since that
>> process was spawned.  So pgpool seems to be in some confused state, as
>> the client definitely severed the connection already.  I checked the
>> pgpool log and there are no explicit references to PID 19838.  I'm at
>> a loss how to debug this further, but clearly something is wrong
>> somewhere, and this isn't normal/expected behavior.
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

Re: [Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-09 Thread Lonni J Friedman 
No one else has experienced this or has suggestions how to debug it?

On Wed, Sep 7, 2011 at 12:49 PM, Lonni J Friedman  wrote:
> Greetings,
> I'm running pgpool-3.0.4 on a Linux-x86_64 server serving as a load
> balancer for a three server postgresql-9.0.4 cluster (1 master, 2
> standby).  I'm seeing strange behavior where a single pgpool process
> seems to hang after some period of time, and then consume 100% of the
> CPU.  I've seen this behavior happen twice since last Friday (when
> pgpool was brought online in my production environment).  At the
> moment the current hung process looks like this in 'ps auxww' output:
>
> postgres 19838 98.7  0.0  68856  2904 ?        R    Sep06 1027:36
> pgpool: lfriedman nightly 10.31.45.20(58277) idle
>
>
> In top, I see:
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 19838 postgres  20   0 68856 2904 1072 R 100.0  0.0   1027:29 pgpool
>
>
> When to connect to the process with strace, there is no output, so I'm
> guessing the process is stuck spinning somewhere:
> # strace -p 19838
> Process 19838 attached - interrupt to quit
> ...
> ^CProcess 19838 detached
>
> One thing that i'm certain of is that the client IP (10.31.45.20)
> associated with the hung process has rebooted at least once since that
> process was spawned.  So pgpool seems to be in some confused state, as
> the client definitely severed the connection already.  I checked the
> pgpool log and there are no explicit references to PID 19838.  I'm at
> a loss how to debug this further, but clearly something is wrong
> somewhere, and this isn't normal/expected behavior.
>
> Help?!
>
> thanks
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general

[Pgpool-general] seemingly hung pgpool process consuming 100% CPU

2011-09-07 Thread Lonni J Friedman 
Greetings,
I'm running pgpool-3.0.4 on a Linux-x86_64 server serving as a load
balancer for a three server postgresql-9.0.4 cluster (1 master, 2
standby).  I'm seeing strange behavior where a single pgpool process
seems to hang after some period of time, and then consume 100% of the
CPU.  I've seen this behavior happen twice since last Friday (when
pgpool was brought online in my production environment).  At the
moment the current hung process looks like this in 'ps auxww' output:

postgres 19838 98.7  0.0  68856  2904 ?RSep06 1027:36
pgpool: lfriedman nightly 10.31.45.20(58277) idle


In top, I see:
  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
19838 postgres  20   0 68856 2904 1072 R 100.0  0.0   1027:29 pgpool


When to connect to the process with strace, there is no output, so I'm
guessing the process is stuck spinning somewhere:
# strace -p 19838
Process 19838 attached - interrupt to quit
...
^CProcess 19838 detached

One thing that i'm certain of is that the client IP (10.31.45.20)
associated with the hung process has rebooted at least once since that
process was spawned.  So pgpool seems to be in some confused state, as
the client definitely severed the connection already.  I checked the
pgpool log and there are no explicit references to PID 19838.  I'm at
a loss how to debug this further, but clearly something is wrong
somewhere, and this isn't normal/expected behavior.

Help?!

thanks
___
Pgpool-general mailing list
Pgpool-general@pgfoundry.org
http://pgfoundry.org/mailman/listinfo/pgpool-general