Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Bruce Momjian wrote:
Tom Lane wrote:
Gregory Stark st...@enterprisedb.com writes:
Tom Lane t...@sss.pgh.pa.us writes:
Ugh. So apparently, we actually need to special-case Solaris to not
believe that posix_fadvise works, or we'll waste cycles uselessly
calling a do-nothing function. Thanks, Sun.
Do we? Or do we just document that setting effective_cache_size on Solaris
won't help?
I assume you meant effective_io_concurrency. We'd still need a special
case because the default is currently hard-wired at 1, not 0, if
configure thinks the function exists. Also there's a posix_fadvise call
in xlog.c that that parameter doesn't control anyhow.
The attached patch prevents the posix_fadvise() probe in configure on
Solaris, and adds a comment why. I have already documented why Solaris
can't do effective_io_concurrency.
Updated patch applied; open item removed as complete.
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: configure
===
RCS file: /cvsroot/pgsql/configure,v
retrieving revision 1.635
diff -c -c -r1.635 configure
*** configure 4 Apr 2009 21:55:49 - 1.635
--- configure 7 Apr 2009 22:45:59 -
***
*** 16324,16331
!
! for ac_func in cbrt dlopen fcvt fdatasync getpeereid getpeerucred getrlimit memmove poll posix_fadvise pstat readlink setproctitle setsid sigprocmask symlink sysconf towlower utime utimes waitpid wcstombs
do
as_ac_var=`echo ac_cv_func_$ac_func | $as_tr_sh`
{ echo $as_me:$LINENO: checking for $ac_func 5
--- 16324,16330
! for ac_func in cbrt dlopen fcvt fdatasync getpeereid getpeerucred getrlimit memmove poll pstat readlink setproctitle setsid sigprocmask symlink sysconf towlower utime utimes waitpid wcstombs
do
as_ac_var=`echo ac_cv_func_$ac_func | $as_tr_sh`
{ echo $as_me:$LINENO: checking for $ac_func 5
***
*** 16419,16427
done
! { echo $as_me:$LINENO: checking whether fdatasync is declared 5
! echo $ECHO_N checking whether fdatasync is declared... $ECHO_C 6; }
! if test ${ac_cv_have_decl_fdatasync+set} = set; then
echo $ECHO_N (cached) $ECHO_C 6
else
cat conftest.$ac_ext _ACEOF
--- 16418,16434
done
! # posix_fadvise() is a no-op on Solaris, so don't incur function overhead
! # by calling it, 2009-04-02
! # http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c
! if test $PORTNAME != solaris; then
!
! for ac_func in posix_fadvise
! do
! as_ac_var=`echo ac_cv_func_$ac_func | $as_tr_sh`
! { echo $as_me:$LINENO: checking for $ac_func 5
! echo $ECHO_N checking for $ac_func... $ECHO_C 6; }
! if { as_var=$as_ac_var; eval test \\${$as_var+set}\ = set; }; then
echo $ECHO_N (cached) $ECHO_C 6
else
cat conftest.$ac_ext _ACEOF
***
*** 16430,16442
cat confdefs.h conftest.$ac_ext
cat conftest.$ac_ext _ACEOF
/* end confdefs.h. */
! #include unistd.h
int
main ()
{
! #ifndef fdatasync
! (void) fdatasync;
#endif
;
--- 16437,16539
cat confdefs.h conftest.$ac_ext
cat conftest.$ac_ext _ACEOF
/* end confdefs.h. */
! /* Define $ac_func to an innocuous variant, in case limits.h declares $ac_func.
!For example, HP-UX 11i limits.h declares gettimeofday. */
! #define $ac_func innocuous_$ac_func
!
! /* System header to define __stub macros and hopefully few prototypes,
! which can conflict with char $ac_func (); below.
! Prefer limits.h to assert.h if __STDC__ is defined, since
! limits.h exists even on freestanding compilers. */
!
! #ifdef __STDC__
! # include limits.h
! #else
! # include assert.h
! #endif
!
! #undef $ac_func
!
! /* Override any GCC internal prototype to avoid an error.
!Use char because int might match the return type of a GCC
!builtin and then its argument prototype would still apply. */
! #ifdef __cplusplus
! extern C
! #endif
! char $ac_func ();
! /* The GNU C library defines this for functions which it implements
! to always fail with ENOSYS. Some functions are actually named
! something starting with __ and the normal name is an alias. */
! #if defined __stub_$ac_func || defined __stub___$ac_func
! choke me
! #endif
int
main ()
{
! return $ac_func ();
! ;
! return 0;
! }
! _ACEOF
! rm -f conftest.$ac_objext conftest$ac_exeext
! if { (ac_try=$ac_link
! case (($ac_try in
! *\* | *\`* | *\\*) ac_try_echo=\$ac_try;;
! *) ac_try_echo=$ac_try;;
! esac
! eval echo \\$as_me:$LINENO: $ac_try_echo\) 5
! (eval $ac_link) 2conftest.er1
! ac_status=$?
! grep -v '^ *+' conftest.er1 conftest.err
! rm -f conftest.er1
! cat conftest.err 5
! echo $as_me:$LINENO: \$? = $ac_status 5
! (exit $ac_status); } {
! test -z $ac_c_werror_flag ||
! test ! -s conftest.err
!
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Tom Lane wrote: Gregory Stark st...@enterprisedb.com writes: Tom Lane t...@sss.pgh.pa.us writes: Ugh. So apparently, we actually need to special-case Solaris to not believe that posix_fadvise works, or we'll waste cycles uselessly calling a do-nothing function. Thanks, Sun. Do we? Or do we just document that setting effective_cache_size on Solaris won't help? I assume you meant effective_io_concurrency. We'd still need a special case because the default is currently hard-wired at 1, not 0, if configure thinks the function exists. Also there's a posix_fadvise call in xlog.c that that parameter doesn't control anyhow. The attached patch prevents the posix_fadvise() probe in configure on Solaris, and adds a comment why. I have already documented why Solaris can't do effective_io_concurrency. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + Index: configure.in === RCS file: /cvsroot/pgsql/configure.in,v retrieving revision 1.592 diff -c -c -r1.592 configure.in *** configure.in 27 Mar 2009 19:58:11 - 1.592 --- configure.in 2 Apr 2009 21:23:36 - *** *** 1141,1150 AC_FUNC_ACCEPT_ARGTYPES PGAC_FUNC_GETTIMEOFDAY_1ARG ! AC_CHECK_FUNCS([cbrt dlopen fcvt fdatasync getpeereid getpeerucred getrlimit memmove poll posix_fadvise pstat readlink setproctitle setsid sigprocmask symlink sysconf towlower utime utimes waitpid wcstombs]) ! AC_CHECK_DECLS(fdatasync, [], [], [#include unistd.h]) AC_CHECK_DECLS(posix_fadvise, [], [], [#include fcntl.h]) AC_CHECK_DECLS([strlcat, strlcpy]) # This is probably only present on Darwin, but may as well check always AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include fcntl.h]) --- 1141,1157 AC_FUNC_ACCEPT_ARGTYPES PGAC_FUNC_GETTIMEOFDAY_1ARG ! AC_CHECK_FUNCS([cbrt dlopen fcvt fdatasync getpeereid getpeerucred getrlimit memmove poll pstat readlink setproctitle setsid sigprocmask symlink sysconf towlower utime utimes waitpid wcstombs]) ! # posix_fadvise() is a no-op on Solaris, so don't incur function overhead ! # by calling it, 2009-04-02 ! # http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c ! if test $PORTNAME != solaris; then ! AC_CHECK_FUNCS(posix_fadvise); AC_CHECK_DECLS(posix_fadvise, [], [], [#include fcntl.h]) + fi + + AC_CHECK_DECLS(fdatasync, [], [], [#include unistd.h]) AC_CHECK_DECLS([strlcat, strlcpy]) # This is probably only present on Darwin, but may as well check always AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include fcntl.h]) -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Robert Haas robertmh...@gmail.com writes: On Fri, Mar 13, 2009 at 10:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: I assume you meant effective_io_concurrency. We'd still need a special case because the default is currently hard-wired at 1, not 0, if configure thinks the function exists. I think 1 should mean no prefetching, rather than 0. No, 1 means prefetch a single block ahead. It doesn't involve I/O concurrency in the sense of multiple I/O requests being processed at once; what it does give you is CPU vs I/O concurrency. 0 shuts that down and returns the system to pre-8.4 behavior. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Jignesh K. Shah j.k.s...@sun.com writes: Can we do a vote on which specific performance features we want to test? Many of the improvements may not be visible through this standard tests so feedback on testing methology for those is also appreciated. * Visibility map - Reduce Vacuum overhead - (I think I can time vacuum with some usage on both databases) Timing vacuum is kind of pointless -- the only thing that matters is whether it's fast enough. But it is worth saying that good benchmarks should include normal vacuum runs. Benchmarks which don't run long enough to trigger vacuum aren't realistic. * Prefetch IO with posix_fadvice () - Though I am not sure if it is supported on UNIX or not (but can be tested by standard tests) Well clearly this is my favourite :) AFAIK Opensolaris doesn't implement posix_fadvise() so there's no benefit. It would be great to hear if you could catch the ear of the right people to get an implementation committed. Depending on how the i/o scheduler system is written it might not even be hard -- the Linux implementation of WILLNEED is all of 20 lines. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication support! -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
A minute ago I said: AFAIK Opensolaris doesn't implement posix_fadvise() so there's no benefit. It would be great to hear if you could catch the ear of the right people to get an implementation committed. Depending on how the i/o scheduler system is written it might not even be hard -- the Linux implementation of WILLNEED is all of 20 lines. I noticed after sending it that that's slightly unfair. The 20-line function calls another function (which calls another function) to do the real readahead work. That function (mm/readahead.c:__do_page_cache_readahead()) is 48 lines. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication support! -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Gregory Stark wrote: A minute ago I said: AFAIK Opensolaris doesn't implement posix_fadvise() so there's no benefit. It would be great to hear if you could catch the ear of the right people to get an implementation committed. Depending on how the i/o scheduler system is written it might not even be hard -- the Linux implementation of WILLNEED is all of 20 lines. I noticed after sending it that that's slightly unfair. The 20-line function calls another function (which calls another function) to do the real readahead work. That function (mm/readahead.c:__do_page_cache_readahead()) is 48 lines. It's implemented. I'm guessing it's not what you want to see though: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Alan Stange sta...@rentec.com writes: Gregory Stark wrote: AFAIK Opensolaris doesn't implement posix_fadvise() so there's no benefit. It's implemented. I'm guessing it's not what you want to see though: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c Ugh. So apparently, we actually need to special-case Solaris to not believe that posix_fadvise works, or we'll waste cycles uselessly calling a do-nothing function. Thanks, Sun. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Tom Lane t...@sss.pgh.pa.us writes: Alan Stange sta...@rentec.com writes: Gregory Stark wrote: AFAIK Opensolaris doesn't implement posix_fadvise() so there's no benefit. It's implemented. I'm guessing it's not what you want to see though: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c Ugh. So apparently, we actually need to special-case Solaris to not believe that posix_fadvise works, or we'll waste cycles uselessly calling a do-nothing function. Thanks, Sun. Do we? Or do we just document that setting effective_cache_size on Solaris won't help? I'm leaning towards the latter because I expect Sun will implement this and there will be people running 8.4 on newer versions of the OS long after it's out. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Gregory Stark st...@enterprisedb.com writes: Tom Lane t...@sss.pgh.pa.us writes: Ugh. So apparently, we actually need to special-case Solaris to not believe that posix_fadvise works, or we'll waste cycles uselessly calling a do-nothing function. Thanks, Sun. Do we? Or do we just document that setting effective_cache_size on Solaris won't help? I assume you meant effective_io_concurrency. We'd still need a special case because the default is currently hard-wired at 1, not 0, if configure thinks the function exists. Also there's a posix_fadvise call in xlog.c that that parameter doesn't control anyhow. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
On Fri, Mar 13, 2009 at 10:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: Gregory Stark st...@enterprisedb.com writes: Tom Lane t...@sss.pgh.pa.us writes: Ugh. So apparently, we actually need to special-case Solaris to not believe that posix_fadvise works, or we'll waste cycles uselessly calling a do-nothing function. Thanks, Sun. Do we? Or do we just document that setting effective_cache_size on Solaris won't help? I assume you meant effective_io_concurrency. We'd still need a special case because the default is currently hard-wired at 1, not 0, if configure thinks the function exists. Also there's a posix_fadvise call in xlog.c that that parameter doesn't control anyhow. I think 1 should mean no prefetching, rather than 0. If the number of concurrent I/O requests was 0, that would mean you couldn't perform any I/O at all. ...Robert -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] 8.4 Performance improvements: was Re: Proposal of tunable fix for scalability of 8.4
Robert Haas robertmh...@gmail.com writes: On Fri, Mar 13, 2009 at 10:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: I assume you meant effective_io_concurrency. We'd still need a special case because the default is currently hard-wired at 1, not 0, if configure thinks the function exists. Also there's a posix_fadvise call in xlog.c that that parameter doesn't control anyhow. I think 1 should mean no prefetching, rather than 0. If the number of concurrent I/O requests was 0, that would mean you couldn't perform any I/O at all. That is actually how I had intended it but apparently I messed it up at some point such that later patches were doing some prefetching at 1 and there was no way to disable it. When Tom reviewed it he corrected the inability to disable prefetching by making 0 disable prefetching. I didn't think it was worth raising as an issue but I didn't realize we were currently doing prefetching by default? i didn't realize that. Even on a system with posix_fadvise there's nothing much to be gained unless the data is on a RAID device, so the original objection holds anyways. We shouldn't do any prefetching unless the user tells us to. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
