Re: [HACKERS] [BUGS] BUG #7521: Cannot disable WAL log while using pg_dump
On Fri, Sep 7, 2012 at 11:40 AM, Gezeala M. Bacuño II wrote:
> adding pgsql-bugs list in case OP posts back.
>
> On Fri, Sep 7, 2012 at 11:29 AM, Pavan Deolasee
> wrote:
>> (Adding -hackers. Did not realize it got dropped)
>>
>> On Fri, Sep 7, 2012 at 11:25 PM, Gezeala M. Bacuño II
>> wrote:
>>>
>>> On Fri, Sep 7, 2012 at 7:17 AM, Pavan Deolasee
>>> wrote:
>>> >
>>> >
>>> > On Fri, Sep 7, 2012 at 7:00 PM, Marie Bacuno II
>>> > wrote:
>>> >>
>>> >>
>>> >> On Sep 7, 2012, at 2:19, Pavan Deolasee
>>> >> wrote:
>>> >>
>>> >>
>>> >> > or have long running transactions ?
>>> >>
>>> >> Yes but I don't think there are when the snapshot was taken. Does the
>>> >> pg_xlog_location_diff() result from latest and prior checkpoint upon
>>> >> start-up indicates the size of replayed changes?
>>> >>
>>> >
>>> > Thats the amount of additional WAL generated after you started the
>>> > server.
>>> >
>>> >>
>>> >> >
>>> >> > BTW, the following query returns ~60GB. Thats the amount of WAL
>>> >> > written after the server was started and at the end of pg_dump (I
>>> >> > don't think pg_xlog_location_diff() is available in the older
>>> >> > releases).
>>> >> >
>>> >> > postgres=# select pg_xlog_location_diff('4450/7A14F280',
>>> >> > '4441/5E681F38')/(2^30);
>>> >> >?column?
>>> >> > --
>>> >> > 60.1980484202504
>>> >>
>>> >> It'll be great to know what the wals modified..?
>>> >
>>> >
>>> > You would need something like xlogdump to decipher them. I quickly tried
>>> > this and it seems to work against 8.4 version that you are running.
>>> > https://github.com/snaga/xlogdump
>>> >
>>> > Download the source code, compile and run it against one of the most
>>> > recent
>>> > WAL files in the cluster against which you ran pg_dump. You would need
>>> > to
>>> > set PATH to contain the pg_config of the server you are running. Please
>>> > post
>>> > the output.
>>> >
>>> > Thanks,
>>> > Pavan
>>> >
>>> >
>>>
>>> Here you go:
>>>
>>> ## last WAL
>>> $ xlogdump -S /dbpool/data/pg_xlog/00014450007A
>>>
>>> /dbpool/data/pg_xlog/00014450007A:
>>>
>>> Unexpected page info flags 0003 at offset 0
>>> Skipping unexpected continuation record at offset 0
>>> ReadRecord: record with zero len at 17488/7A14F310
>>> Unexpected page info flags 0001 at offset 15
>>> Skipping unexpected continuation record at offset 15
>>> Unable to read continuation page?
>>> ** maybe continues to next segment **
>>> ---
>>> TimeLineId: 1, LogId: 17488, LogSegment: 122
>>>
>>> Resource manager stats:
>>> [0]XLOG : 3 records, 120 bytes (avg 40.0 bytes)
>>> checkpoint: 3, switch: 0, backup end: 0
>>> [1]Transaction: 0 record, 0 byte (avg 0.0 byte)
>>> commit: 0, abort: 0
>>> [2]Storage : 0 record, 0 byte (avg 0.0 byte)
>>> [3]CLOG : 0 record, 0 byte (avg 0.0 byte)
>>> [4]Database : 0 record, 0 byte (avg 0.0 byte)
>>> [5]Tablespace: 0 record, 0 byte (avg 0.0 byte)
>>> [6]MultiXact : 0 record, 0 byte (avg 0.0 byte)
>>> [7]Reserved 7: 0 record, 0 byte (avg 0.0 byte)
>>> [8]Reserved 8: 0 record, 0 byte (avg 0.0 byte)
>>> [9]Heap2 : 2169 records, 43380 bytes (avg 20.0 bytes)
>>> [10]Heap : 0 record, 0 byte (avg 0.0 byte)
>>> ins: 0, upd/hot_upd: 0/0, del: 0
>>> [11]Btree : 0 record, 0 byte (avg 0.0 byte)
>>> [12]Hash : 0 record, 0 byte (avg 0.0 byte)
>>> [13]Gin : 0 record, 0 byte (avg 0.0 byte)
>>> [14]Gist : 0 record, 0 byte (avg 0.0 byte)
>>> [15]Sequence : 0 record, 0 byte (avg 0.0 byte)
>>>
>>> Backup block stats: 2169 blocks, 16551816 bytes (avg 7631.1 bytes)
>>>
>>
>> I think both my theories seem to be holding up. Heap2 resource manager is
>> used only for vacuum freeze, lazy vacuum or HOT prune. Given your access
>> pattern, I bet its the third activity that kicking in on your database. You
>> got many pages with dead tuples and they are getting cleaned at the first
>> opportunity, which happens to be the pg_dump thats run immediately after the
>> server restart. This is seen by all 2169 WAL records in the file being
>> attributed to the Heap2 RM above.
>>
>> Whats additionally happening is each of these records are on different heap
>> pages. The cleanup activity dirties those pages. Since each of these pages
>> is being dirtied for the first time after a recent checkpoint and
>> full_page_writes is turned ON, entire page is backed up in the WAL record.
>> You can see the exact number of backup blocks in the stats above.
>>
>> I don't think we have any mechanism to control or stop HOT from doing what
>> it wants to do, unless you are willing to run a modified server for this
>> reason. But you can at least bring down the WAL volume by turning
>> full_page_writes OFF.
>>
>> Thanks,
>> Pavan
>
> Great. Finally got some light on this. I'll disable full_page_writes
> on my next backup and will post back results tomorrow. Than
Re: [HACKERS] Supporting plpython 2+3 builds better
On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
> To give you an idea of what "unreasonably painful" means, attached is
> the specfile diff needed to make this happen. I will not comment on
> the fragility of this beyond observing that the "touch -r" commands
> are *necessary*, else you get incorrect results.
I think an easier way is to configure two vpath builds and install the
pieces you want from each one. yo
> Another problem is that Makefile.shlib isn't designed to build more
> than one shared library per directory,
That's the main problem, but fixing it would be very useful in other
places as well. I had it on my radar to do something about that.
> which means that we might end up having to copy all the source files
> into a new plpython3 subdirectory anyway. We're already doing some of
> that with the regression test files, though.
Doing it with the test files is already annoying enough. For instance,
we don't have a static list of all the files we need to copy (because of
expected file variants that are not tracked in the makefiles), so we
need to copy the files again on each run. If you extend this to all
source files, things would get totally bizarre.
A less invasive method might be to set up a second directory
pl/plpython3 which vpath-builds from pl/plpython.
But frankly, I'd like to work on the multiple-shlibs-in-one-directory
issue, because all of the above issues and more are projected in their
own strange ways on something like the transforms feature. For example,
how would you organize the source code for an extension module that
provides a new data type and adapters for plpython{2,3}u and plperl{,u},
with tests and all that?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote: > Anyway, what I notice is that I get different types of failures, but > they are all under ecpg/. What I think we need to do is insert > .NOTPARALLEL in ecpg/Makefile, I'd hate that, because the ecpg build is one of the slowest parts of the build, so de-parallelizing it would slow down everything quite a bit. > because there are several reasons not > to run its sub-makes in parallel: > > * preproc/Makefile casually does this: > > ../ecpglib/typename.o: ../ecpglib/typename.c > $(MAKE) -C $(dir $@) $(notdir $@) > > which is very likely to screw up any make proceeding in parallel in > ecpglib. That should probably be fixed by symlinking the source file and building it in the preproc directory. > And that's not even counting the bison-output problem you were seeing. > I'm not entirely sure what's causing that, but I'm suspicious that the > ultimate cause is the extra rules for the "all...recurse" targets in > ecpg/Makefile, which look like they could result in additional instances > of multiple make processes running in the same subdirectory. I think the point of these targets is exactly to prevent that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python version dependency in plpython regression tests
On Sat, 2012-09-08 at 16:52 -0400, Tom Lane wrote: > How come you did not back-patch that commit ... are we not supporting > 3.3 in branches before 9.2 for some reason? Python 3.3 isn't even released yet, much less so back then, so it seemed premature. Also, it's a fairly big change just to make the regression tests pass. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On 09/08/2012 07:54 PM, Tom Lane wrote: Andrew Dunstan writes: I have just repeated this on an absolutely fresh up to date F17 machine, with no symlink stuff in play. Steps to recreate: CC="ccache gcc" ../postgres/configure --enable-depend --enable-debug --enable-cassert --with-perl --with-python --with-tcl --with-libxml --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap make -j 4 Huh ... that recipe works (er, fails) for me too, at least some of the time. I wonder what exactly is the key difference between the working and failing cases? Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, because there are several reasons not to run its sub-makes in parallel: * preproc/Makefile casually does this: ../ecpglib/typename.o: ../ecpglib/typename.c $(MAKE) -C $(dir $@) $(notdir $@) which is very likely to screw up any make proceeding in parallel in ecpglib. * compatlib and ecpglib will equally casually invoke "make all" in other subdirectories; ditto. And that's not even counting the bison-output problem you were seeing. I'm not entirely sure what's causing that, but I'm suspicious that the ultimate cause is the extra rules for the "all...recurse" targets in ecpg/Makefile, which look like they could result in additional instances of multiple make processes running in the same subdirectory. After adding the .NOTPARALLEL marker, I don't see these failures anymore. Well, I'm glad it's not just me. :-) This fix works for me too. I guess it should be applied back to 9.1, when it looks like we started using .NOTPARALLEL cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
Andrew Dunstan writes: > I have just repeated this on an absolutely fresh up to date F17 machine, > with no symlink stuff in play. > Steps to recreate: > CC="ccache gcc" ../postgres/configure --enable-depend --enable-debug > --enable-cassert --with-perl --with-python --with-tcl --with-libxml > --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap > make -j 4 Huh ... that recipe works (er, fails) for me too, at least some of the time. I wonder what exactly is the key difference between the working and failing cases? Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, because there are several reasons not to run its sub-makes in parallel: * preproc/Makefile casually does this: ../ecpglib/typename.o: ../ecpglib/typename.c $(MAKE) -C $(dir $@) $(notdir $@) which is very likely to screw up any make proceeding in parallel in ecpglib. * compatlib and ecpglib will equally casually invoke "make all" in other subdirectories; ditto. And that's not even counting the bison-output problem you were seeing. I'm not entirely sure what's causing that, but I'm suspicious that the ultimate cause is the extra rules for the "all...recurse" targets in ecpg/Makefile, which look like they could result in additional instances of multiple make processes running in the same subdirectory. After adding the .NOTPARALLEL marker, I don't see these failures anymore. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On 09/08/2012 04:52 PM, Andrew Dunstan wrote: On 09/08/2012 04:46 PM, Tom Lane wrote: Andrew Dunstan writes: Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. No. just "make -j 4" And it's the stock Fedora build of make. I have just repeated this on an absolutely fresh up to date F17 machine, with no symlink stuff in play. Steps to recreate: CC="ccache gcc" ../postgres/configure --enable-depend --enable-debug --enable-cassert --with-perl --with-python --with-tcl --with-libxml --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap make -j 4 cheers andrew q -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Supporting plpython 2+3 builds better
I wrote:
> After reading the recent thread about python 2 vs python 3 support,
> I thought I'd amuse myself by trying to get plpython3 supported in
> the Fedora packages. That turned out to be unreasonably painful
> (which is something we oughta fix eventually), but it worked,
To give you an idea of what "unreasonably painful" means, attached is
the specfile diff needed to make this happen. I will not comment on
the fragility of this beyond observing that the "touch -r" commands
are *necessary*, else you get incorrect results.
I think we really need to do something to make this type of build less
painful, because as far as I can see most distros are going to be
wanting to support both python 2 and 3 for awhile to come.
A sketch of a solution might go like this:
* Extend configure to have two switches, say "--with-python" and
"--with-python3", which you can select either or both of. If you
want to pick executables that are not named "python" and "python3",
perhaps write it like "--with-python3=/usr/bin/python3.2mu". (I'd
be inclined to remove the dependency on a PYTHON envvar altogether.)
* Make configure output two independent sets of variables into
Makefile.global, viz
python2_includespec = ...
python2_libdir = ...
python2_libspec = ...
python2_additional_libs = ...
python2_configdir = ...
python2_majorversion= ...
python2_version = ...
python3_includespec = ...
python3_libdir = ...
python3_libspec = ...
python3_additional_libs = ...
python3_configdir = ...
python3_majorversion= ...
python3_version = ...
* Make plpython's Makefile build, test, install plpython2 and/or
plpython3 depending on what's set in Makefile.global.
I'm not sure yet whether it's possible to do this without duplicating a
lot of logic in config/python.m4 and plpython's Makefile.
Another problem is that Makefile.shlib isn't designed to build more than
one shared library per directory, which means that we might end up
having to copy all the source files into a new plpython3 subdirectory
anyway. We're already doing some of that with the regression test
files, though.
Thoughts?
regards, tom lane
diff --git a/postgresql.spec b/postgresql.spec
index 524d81c..62ce352 100644
*** a/postgresql.spec
--- b/postgresql.spec
***
*** 35,40
--- 35,41
%{!?test:%global test 1}
%{!?upgrade:%global upgrade 1}
%{!?plpython:%global plpython 1}
+ %{!?plpython3:%global plpython3 1}
%{!?pltcl:%global pltcl 1}
%{!?plperl:%global plperl 1}
%{!?ssl:%global ssl 1}
*** BuildRequires: systemd-units
*** 107,112
--- 108,117
BuildRequires: python-devel
%endif
+ %if %plpython3
+ BuildRequires: python3-devel
+ %endif
+
%if %pltcl
BuildRequires: tcl-devel
%endif
*** Install this if you want to write databa
*** 265,278
%if %plpython
%package plpython
! Summary: The Python procedural language for PostgreSQL
Group: Applications/Databases
Requires: %{name}-server%{?_isa} = %{version}-%{release}
%description plpython
The postgresql-plpython package contains the PL/Python procedural language,
which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python.
%endif
%if %pltcl
--- 270,295
%if %plpython
%package plpython
! Summary: The Python2 procedural language for PostgreSQL
Group: Applications/Databases
Requires: %{name}-server%{?_isa} = %{version}-%{release}
%description plpython
The postgresql-plpython package contains the PL/Python procedural language,
which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 2.
! %endif
!
! %if %plpython3
! %package plpython3
! Summary: The Python3 procedural language for PostgreSQL
! Group: Applications/Databases
! Requires: %{name}-server%{?_isa} = %{version}-%{release}
!
! %description plpython3
! The postgresql-plpython3 package contains the PL/Python3 procedural language,
! which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 3.
%endif
%if %pltcl
*** CFLAGS="$CFLAGS -DLINUX_OOM_SCORE_ADJ=0"
*** 346,351
--- 363,447
# CFLAGS=`echo $CFLAGS| sed -e "s|-O2|-O1|g" `
# %%endif
+ # plpython requires separate configure/build runs to build against python 2
+ # versus python 3. Our strategy is to do the python 3 run first, then make
+ # distclean and do it again for the "normal" build. Note that the installed
+ # Makefile.global will reflect the python 2 build, which seems appropriate
+ # since that's still considered the default plpython version.
+ %if %plpython3
+
+ export PYTHON=/usr/bin/python3
+
+ # These configure options must match ma
Re: [HACKERS] pg_test_fsync output and commit_delay
> The attached simple patch alters the output produced by pg_test_fsync, > so that we also see the average sync time per op. Good idea. A median would be even better, but harder to calculate, I imagine. You might consider providing a maximum, too. > The pg_test_fsync > docs have had minor alterations too. Hopefully, another doc patch will > follow that builds upon this, and actually firmly recommends taking a > particular course of action when setting commit_delay - my previous > observations about what helped throughput, though supported by Greg > Smith, have not been scrutinised enough just yet, I feel. For now, the > equivocated wording of my doc alterations (that raw wal_sync_method > file sync time might *somehow* be useful here) seems appropriate. I'll see what I can do. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_upgrade testing for MSVC
Here's a patch to provide for the pg_upgrade tests to be run in MSVC
builds. I want to get this running on the buildfarm before long on HEAD
and REL9_2_STABLE.
cheers
andrew
diff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm
index 3923532..235a150 100644
--- a/src/tools/msvc/Install.pm
+++ b/src/tools/msvc/Install.pm
@@ -37,9 +37,16 @@ sub Install
$| = 1;
my $target = shift;
- our $config;
- require "config_default.pl";
- require "config.pl" if (-f "config.pl");
+ # if called from vcregress, the config will be passed to us
+ # so no need to re-include these
+ our $config = shift;
+ unless ($config)
+ {
+ # suppress warning about harmless redeclaration of $config
+ no warnings 'misc';
+ require "config_default.pl";
+ require "config.pl" if (-f "config.pl");
+ }
chdir("../../..")if (-f "../../../configure");
chdir("../../../..") if (-f "../../../../configure");
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 530770c..fed4916 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -9,15 +9,19 @@ our $config;
use Cwd;
use File::Copy;
+use Install qw(Install);
+
my $startdir = getcwd();
chdir "../../.." if (-d "../../../src/tools/msvc");
+my $topdir = getcwd();
+
require 'src/tools/msvc/config_default.pl';
require 'src/tools/msvc/config.pl' if (-f 'src/tools/msvc/config.pl');
# buildenv.pl is for specifying the build environment settings
-# it should contian lines like:
+# it should contain lines like:
# $ENV{PATH} = "c:/path/to/bison/bin;$ENV{PATH}";
if (-e "src/tools/msvc/buildenv.pl")
@@ -27,7 +31,7 @@ if (-e "src/tools/msvc/buildenv.pl")
my $what = shift || "";
if ($what =~
- /^(check|installcheck|plcheck|contribcheck|ecpgcheck|isolationcheck)$/i)
+ /^(check|installcheck|plcheck|contribcheck|ecpgcheck|isolationcheck|upgradecheck)$/i)
{
$what = uc $what;
}
@@ -73,7 +77,8 @@ my %command = (
INSTALLCHECK => \&installcheck,
ECPGCHECK => \&ecpgcheck,
CONTRIBCHECK => \&contribcheck,
- ISOLATIONCHECK => \&isolationcheck,);
+ ISOLATIONCHECK => \&isolationcheck,
+UPGRADECHECK => \&upgradecheck,);
my $proc = $command{$what};
@@ -231,6 +236,76 @@ sub contribcheck
exit $mstat if $mstat;
}
+sub upgradecheck
+{
+ my $status;
+ my $cwd = getcwd();
+
+ # Much of this comes from the pg_upgrade test.sh script,
+ # but it only covers the --install case, and not the case
+ # where the old and new source or bin dirs are different.
+ # i.e. only the this version to this version check. That's
+ # what pg_upgrade's "make check" does.
+
+ $ENV{PGPORT} ||= 50432;
+ my $tmp_root = "$topdir/contrib/pg_upgrade/tmp_check";
+ mkdir $tmp_root unless -d $tmp_root;
+ my $tmp_install = "$tmp_root/install";
+ print "Setting up temp install\n\n";
+ Install($tmp_install, $config);
+ # Install does a chdir, so change back after that
+ chdir $cwd;
+ my ($bindir,$libdir,$oldsrc,$newsrc) =
+ ("$tmp_install/bin", "$tmp_install/lib", $topdir, $topdir);
+ $ENV{PATH} = "$bindir;$ENV{PATH}";
+ my $data = "$tmp_root/data";
+ $ENV{PGDATA} = $data;
+ my $logdir = "$topdir/contrib/pg_upgrade/log";
+ mkdir $logdir unless -d $logdir;
+ print "\nRunning initdb on old cluster\n\n";
+ system("initdb");
+ print "\nStarting old cluster\n\n";
+ system("pg_ctl start -l $logdir/postmaster1.log -w");
+ print "\nSetting up data for upgrading\n\n";
+ installcheck();
+ # now we can chdir into the source dir
+ chdir "$topdir/contrib/pg_upgrade";
+ print "\nDuming old cluster\n\n";
+ system("pg_dumpall -f $tmp_root/dump1.sql");
+ print "\nStopping old cluster\n\n";
+ system("pg_ctl -m fast stop");
+ rename $data, "$data.old";
+ print "\nSetting up new cluster\n\n";
+ system("initdb");
+ print "\nRunning pg_upgrade\n\n";
+ system("pg_upgrade -d $data.old -D $data -b $bindir -B $bindir");
+ print "\nStarting new cluster\n\n";
+ system("pg_ctl -l $logdir/postmaster2.log -w start");
+ print "\nSetting up stats on new cluster\n\n";
+ system(".\\analyze_new_cluster.bat");
+ print "\nDumping new cluster\n\n";
+ system("pg_dumpall -f $tmp_root/dump2.sql");
+ print "\nStopping new cluster\n\n";
+ system("pg_ctl -m fast stop");
+ print "\nDeleting old cluster\n\n";
+ system(".\\delete_old_cluster.bat");
+ print "\nComparing old and new cluster dumps\n\n";
+
+ # If any of the above failed, the following will also fail, so
+ # this should be a sufficient check of the whole thing.
+ system("diff -q $tmp_root/dump1.sql $tmp_root/dump2.sql");
+ $status = $?;
+ if (!$status)
+ {
+ print "PASSED\n";
+ }
+ else
+ {
+ print "dumps not identical!\n";
+ exit(1);
+ }
+}
+
sub fetchRegressOpts
{
my $handle;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_test_fsync output and commit_delay
I propose that we try and develop better commit_delay advice, to make it easier to set the parameter in a way that actually helps performance. I have been researching a way to make commit_delay adaptive, though have yet to develop any further insight that is worth sharing. This may change. The attached simple patch alters the output produced by pg_test_fsync, so that we also see the average sync time per op. The pg_test_fsync docs have had minor alterations too. Hopefully, another doc patch will follow that builds upon this, and actually firmly recommends taking a particular course of action when setting commit_delay - my previous observations about what helped throughput, though supported by Greg Smith, have not been scrutinised enough just yet, I feel. For now, the equivocated wording of my doc alterations (that raw wal_sync_method file sync time might *somehow* be useful here) seems appropriate. -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services pg_test_fsync.v1.2012_09_08.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On 09/08/2012 04:46 PM, Tom Lane wrote: Andrew Dunstan writes: Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. No. just "make -j 4" And it's the stock Fedora build of make. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On 09/08/2012 04:54 PM, Tom Lane wrote: Andrew Dunstan writes: And it's the stock Fedora build of make. Which Fedora branch exactly? The package version I was trying to reproduce with here is make-3.82-8.fc16.x86_64. Same: $ rpm -q make make-3.82-8.fc16.x86_64 cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python version dependency in plpython regression tests
Peter Eisentraut writes: > On Sat, 2012-09-08 at 16:35 -0400, Tom Lane wrote: >> I think probably the best thing is to change the test case so it has >> one valid key and one not-valid one, rather than assuming that the >> same key will always be complained of when there's more than one >> not-valid one. > That would probably work. We dealt with a similar problem in > 2cfb1c6f77734db81b6e74bcae630f93b94f69be, if you want some additional > inspiration. Not sure why we didn't see this case then. If it's got anything to do with hashing, platform dependency wouldn't be a bit surprising. Or they might have tweaked the algorithm some more since May. How come you did not back-patch that commit ... are we not supporting 3.3 in branches before 9.2 for some reason? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
Andrew Dunstan writes: > And it's the stock Fedora build of make. Which Fedora branch exactly? The package version I was trying to reproduce with here is make-3.82-8.fc16.x86_64. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
Andrew Dunstan writes: > Scratch that theory, that was just a transient. If anything it looks > like it is related to system load. When almost nothing was running on > the machine it worked fine. When I started up a Browser and an MUA the > problem occurred. This VM has 4 CPUs and 4Gb of memory and a load > average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python version dependency in plpython regression tests
On Sat, 2012-09-08 at 16:35 -0400, Tom Lane wrote: > and obviously, python is iterating through the hash's keys in a > different order than it was a minor version or two back. (The failure > is occurring with 3.3.0-0.4.rc1.fc19, whereas I saw no failure with > 3.2.3-7.fc17.) Yes, known problem with 3.3. > > I think probably the best thing is to change the test case so it has > one > valid key and one not-valid one, rather than assuming that the same > key > will always be complained of when there's more than one not-valid > one. That would probably work. We dealt with a similar problem in 2cfb1c6f77734db81b6e74bcae630f93b94f69be, if you want some additional inspiration. Not sure why we didn't see this case then. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Python version dependency in plpython regression tests
After reading the recent thread about python 2 vs python 3 support,
I thought I'd amuse myself by trying to get plpython3 supported in
the Fedora packages. That turned out to be unreasonably painful
(which is something we oughta fix eventually), but it worked,
at least with F16/F17. When I went to try to build it in rawhide,
I got these failures from the plpython_trigger regression test:
*** /builddir/build/BUILD/postgresql-9.2.0/src/pl/plpython/./python3/expected/pl
python_trigger.out Sat Sep 8 16:20:55 2012
--- /builddir/build/BUILD/postgresql-9.2.0/src/pl/plpython/./python3/results/plp
ython_trigger.out Sat Sep 8 16:21:07 2012
***
*** 483,489
BEFORE UPDATE ON trigger_test
FOR EACH ROW EXECUTE PROCEDURE stupid7();
UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR: key "a" found in TD["new"] does not exist as a column in the triggerin
g row
CONTEXT: while modifying trigger row
PL/Python function "stupid7"
DROP TRIGGER stupid_trigger7 ON trigger_test;
--- 483,489
BEFORE UPDATE ON trigger_test
FOR EACH ROW EXECUTE PROCEDURE stupid7();
UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR: key "b" found in TD["new"] does not exist as a column in the triggerin
g row
CONTEXT: while modifying trigger row
PL/Python function "stupid7"
DROP TRIGGER stupid_trigger7 ON trigger_test;
***
*** 497,503
BEFORE UPDATE ON trigger_test
FOR EACH ROW EXECUTE PROCEDURE stupid7u();
UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR: key "a" found in TD["new"] does not exist as a column in the triggerin
g row
CONTEXT: while modifying trigger row
PL/Python function "stupid7u"
DROP TRIGGER stupid_trigger7 ON trigger_test;
--- 497,503
BEFORE UPDATE ON trigger_test
FOR EACH ROW EXECUTE PROCEDURE stupid7u();
UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR: key "b" found in TD["new"] does not exist as a column in the triggerin
g row
CONTEXT: while modifying trigger row
PL/Python function "stupid7u"
DROP TRIGGER stupid_trigger7 ON trigger_test;
The test cases being complained of look like
-- TD keys not corresponding to row columns
CREATE FUNCTION stupid7() RETURNS trigger
AS $$
TD["new"] = {'a': 'foo', 'b': 'bar'}
return "MODIFY";
$$ LANGUAGE plpythonu;
and obviously, python is iterating through the hash's keys in a
different order than it was a minor version or two back. (The failure
is occurring with 3.3.0-0.4.rc1.fc19, whereas I saw no failure with
3.2.3-7.fc17.)
I think probably the best thing is to change the test case so it has one
valid key and one not-valid one, rather than assuming that the same key
will always be complained of when there's more than one not-valid one.
Any objections, or different analysis?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
On 09/08/2012 11:06 AM, Tom Lane wrote: Andrew Dunstan writes: This seems totally stupid, but it happens when the path to the current directory includes a cross-device symlink. If I cd following the link, then this effect doesn't happen. Weird. Huh. So maybe a gmake bug, or maybe there's something wrong with our make rules for the case that `pwd` doesn't match what gmake thinks the current path is. Could you specify the setup more fully? Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Question about SSI, subxacts, and aborted read-only xacts
This question comes about after reading the VLDB paper "Serializable Snapshot Isolation in PostgreSQL". We release predicate locks after a transaction abort, but not after a subtransaction abort. The paper says that the reason is: "We do not drop SIREAD locks acquired during a subtransaction if the subtransaction is aborted (i.e. all SIREAD locks belong to the top-level transaction). This is because data read during the subtransaction may have been reported to the user or otherwise externalized." (section 7.3). But that doesn't make sense to me, because that reasoning would also apply to top-level transactions that are aborted, but we release the SIREAD locks for those. In other words, this introduces an inconsistency between: BEGIN ISOLATION LEVEL SERIALIZABLE; SAVEPOINT s1; ... ROLLBACK TO s1; COMMIT; and: BEGIN ISOLATION LEVEL SERIALIZABLE; ... ROLLBACK; I'm not suggesting this is a correctness problem: holding SIREAD locks for longer never causes incorrect results. But it does seem a little inconsistent. For top-level transactions, I don't think it's possible to preserve SIREAD locks after an abort, because we rely on aborts to alleviate conflicts (and when using 2PC, we may need to abort a read-only transaction to correct the situation). So it seems like users must not rely on any answers they get from a transaction (or subtransaction) unless it commits. Does that make sense? If so, I think we need a documentation update. The serializable isolation level docs don't quite make it clear that serializability only applies to transactions that commit. It might not be obvious to a user that there's a difference between commit and abort for a RO transaction. I think that, in S2PL, serializability applies even to aborted transactions (though I haven't spent much time thinking about it), so users accustomed to other truly-serializable implementations might be surprised. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: disable bypass of expand variadic for "ANY" variable function, format_array function for formatting with params in array
2012/9/8 Tom Lane :
> Pavel Stehule writes:
>> This patch disable bypassing of parameters for variadic function with
>> "ANY" type variadic parameter.
>
> This seems completely silly. If you think it's broken now (which I
> don't particularly agree with: "does not do what you want in some corner
> cases" is not "broken") then propose a fix. Breaking it worse is not an
> improvement.
it is broken
format('%s %s", 'Hello', 'World') -- is ok -- case A
format('%s %s', variadic array['Hello', 'World']) -- fails -- case B
Now, there are no possibility detect from function if there is a A
case or B case.
probably there are three fixes:
a) enhance FunctionCallInfoData by "expand_variadic" field - and then
different behave should be implemented in function,
b) enhance executor to use a updated FmgrInfo for every call of
function. FmgrInfo should be updated every call because fn_nargs can
be changed every call,
c) change mechanism how variadic parameters are passed to variadic
"any" function. Now we use FunctionCallInfoData. We can pass only
pointer to some structure with parameters enhanced about type info.
This mechanism can be same for A case and B case. And we can share
FmgrInfo - because there will be only one real parameter of type
internal. But this change is not compatible with current design. But
is a most simple probably and decrease difference between variadic
"any" functions and others variadic functions.
We can inplement a new datatype "any"[] - and this can be flag for new
implementation and "any" for old implementation. So there should not
be problem with compatibility.
Regards
Pavel
>
> regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: disable bypass of expand variadic for "ANY" variable function, format_array function for formatting with params in array
Pavel Stehule writes: > This patch disable bypassing of parameters for variadic function with > "ANY" type variadic parameter. This seems completely silly. If you think it's broken now (which I don't particularly agree with: "does not do what you want in some corner cases" is not "broken") then propose a fix. Breaking it worse is not an improvement. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] patch: disable bypass of expand variadic for "ANY" variable function, format_array function for formatting with params in array
Hello
This patch disable bypassing of parameters for variadic function with
"ANY" type variadic parameter. Now - this functionality is just
broken. Because there are no any requests for fixing this issue, I
propose the most simply solution - just disable using this type of
variadic function when variadic variables are not expanded. Internally
it has impact to "format" function only. There are no possibility put
parameters in array. But this possibility can be useful. I cannot to
use overloading due ambiguous functions with "any" and "anyarray"
params. Solution can be new function "format_array" - that share code
with "format" function.
postgres=# select format_array('Hello %s, %1$s', array['World']);
format_array
Hello World, World
(1 row)
postgres=# select format('Hello %s, %1$s', variadic array['World']);
ERROR: function format(unknown, text[]) does not exist
LINE 1: select format('Hello %s, %1$s', variadic array['World']);
^
HINT: No function matches the given name and argument types. You
might need to add explicit type casts.
postgres=# select format('Hello %s, %1$s', 'World');
format
Hello World, World
(1 row)
Regards
Pavel Stehule
text_format_array.diff
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using mixed results
Andrew Dunstan writes: > This seems totally stupid, but it happens when the path to the current > directory includes a cross-device symlink. If I cd following the link, > then this effect doesn't happen. Weird. Huh. So maybe a gmake bug, or maybe there's something wrong with our make rules for the case that `pwd` doesn't match what gmake thinks the current path is. Could you specify the setup more fully? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol
A Dijous, 6 de setembre de 2012 00:30:53, Josh Berkus va escriure: > > In general I think the selling point for such a feature would be "no > > administrative hassles", and I believe that has to go not only for the > > end-user experience but also for the application-developer experience. > > If you have to manage checkpointing and vacuuming in the application, > > you're probably soon going to look for another database. > > Well, don't discount the development/testing case. If you do agile or > TDD (a lot of people do), you often have a workload which looks like: > > 1) Start framework > 2) Start database > 3) Load database with test data > 4) Run tests > 5) Print results > 6) Shut down database > > In a case like that, you can live without checkpointing, even; the > database is ephemeral. > > In other words, let's make this a feature and document it for use in > testing, and that it's not really usable for production embedded apps yet. +1. Some projects such as tryton would benefit from this feature. -- Albert Cervera i Areny http://www.NaN-tic.com Tel: +34 93 553 18 03 http://twitter.com/albertnan http://www.nan-tic.com/blog -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
